Amazon’s S3 web-based storage service is experiencing widespread issues, leading to service that’s either partially or fully broken on websites, apps and devices upon which it relies. The AWS offering provides hosting for images for a lot of sites, and also hosts entire websites, and app backends including Nest.
The S3 outage is due to “high error rates with S3 in US-EAST-1,” according to Amazon’s AWS service health dashboard, which is where the company also says it’s working on “remediating the issue,” without initially revealing any further details.
Affected websites and services include Quora, newsletter provider Sailthru, Business Insider, Giphy, image hosting at a number of publisher websites, filesharing in Slack, and many more. Connected lightbulbs, thermostats and other IoT hardware is also being impacted, with many unable to control these devices as a result of the outage.
Amazon S3 is used by around 148,213 websites, and 121,761 unique domains, according to data tracked by SimilarTech, and its popularity as a content host concentrates specifically in the U.S. It’s used by 0.8 percent of the top 1 million websites, which is actually quite a bit smaller than CloudFlare, which is used by 6.2 percent of the top 1 million websites globally – and yet it’s still having this much of an effect.
Amazingly, even the status indicators on the AWS service status page rely on S3 for storage of its health marker graphics, hence why the site is still showing all services green despite obvious evidence to the contrary.
The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.
— Amazon Web Services (@awscloud) February 28, 2017
We’re monitoring the situation and will provide more info as it becomes available.
Update (11:40 AM PT): AWS has fixed the issues with its own dashboard at least – it’ll now accurately reflect service status as it continues to attempt to fix the problem.
Update (11:57 AM PT): AWS says it believes they new “understand root cause” of the S3 issues, and are “working hard at repairing.” It has not shared specifics of that cause.
Update (12:15 PM PT): Network intelligence software provider ThousandEyes notes that all the packet loss for the ongoing issue appears to be happening in the Ashburn, VA area. Amazon has an AWS data center in Ashburn, whose exact location was revealed in a news story last year due to a fire during its construction.
Update (12:54 PM PT): AWS says it’s seeing “recovery for S3 object retrievals, listing and deletions” which means you’re probably seeing avatars and other visuals assets come back in some spots. The company also says it expects further improvements to error rates within the next hour.
Update (1:20 PM PT): S3 is now fully recovered in terms of the retrieval, listing and deletion of existing objects, according to the AWS status page, and it’s now working on restoring normal operation for the addition of new items to S3-based storage.
Update (2:10 PM PT): AWS says that it’s now fully recovered in terms of resolving the error rates it was seeing, and S3 service is now “operating normally.”
Comment