On the night of May 7th into the 8th, 2026, one availability zone in Northern Virginia got too hot.
That’s the actual root cause, put plainly. AWS called it a “thermal event.” Temperatures climbed in a single zone of us-east-1, the one with the boring internal name use1-az4, hot enough to impair EC2 instances and start degrading the EBS volumes underneath them. One zone. Not a region, not AWS as a whole, not some giant multi-continent pile-up of the kind that makes the news for a week. And in the seven-hour window from midnight to 7am, StatusGator logged 2,370 status changes across the 748 services it watches, with more than 150 distinct products going visibly, customer-facing down.
Here’s the part worth actually sitting with. Almost every one of those 150 services would have told you, the Friday before, that they were multi-AZ.
Multi-AZ on the slide, single-AZ in the data path
Redundancy that has never failed over isn’t redundancy. It’s a diagram.
I’ve reviewed enough architectures to know this pattern in my sleep, and it always goes the same way. On paper it’s perfect: the deck says active-active across three zones, the Terraform says three zones, the load balancer really is spread across all three. And then you follow one real request all the way down to the bottom and find that the primary Postgres lives in use1-az4 with a replica that’s technically in az2 but has never once been promoted under real load, or the promotion is manual and the runbook for it is a Confluence page last edited in 2024 by an engineer who has since left the company, or the “multi-AZ” service quietly depends on a config store that itself only runs in one zone, or, my personal favorite, the failover mechanism works perfectly except the other two zones don’t have enough spare capacity to actually take the traffic, so they tip over in turn about ninety seconds after they take it on. Every layer looked redundant in isolation. A single zone ran straight through the data path the entire time, and nobody caught it, because nobody had ever pulled the thread.
That’s what the 8th exposed. Not that AWS is fragile. AWS did exactly what it says on the tin, it lost a zone and kept the other two serving. Every team that went dark had been silently depending on one specific zone while genuinely believing they weren’t.
Why the failover you paid for doesn’t fire
You are almost certainly paying for redundancy at this exact moment. Second-zone replicas, standby capacity, cross-zone data transfer charges, all of it. It’s a real, recurring line on your bill.
The problem is that redundancy is a mechanism, and mechanisms that never actually run will rust in place where they sit. Health checks drift out of sync with what’s really healthy. A security-group change three months back quietly severed the network path the failover depends on, and no alert caught it because the path was never used, so it sat there broken and invisible. DNS TTLs are set to values somebody picked in 2022 that mean your traffic takes a full eleven minutes to physically move even after every other piece works perfectly. The standby database has fallen far enough behind on replication that promoting it means accepting data loss, so your on-call engineer hesitates at 2am, which is genuinely the correct instinct and also the exact reason the outage runs an extra hour. None of this surfaces until the day the zone actually goes dark, and by then you’re learning it live, in production, with customers watching, which is the single most expensive classroom on earth for discovering your disaster recovery was only ever a guess.
And the money is not small. The Splunk and Cisco “Hidden Costs of Downtime” report for 2026 puts the average at over $15,000 a minute once you count all the knock-on damage: lost transactions, SLA credits, the engineers yanked off roadmap work for a week, the churn that doesn’t show up on a dashboard until next quarter. Their headline number is $600 billion a year in downtime losses across the Global 2000, up from $400 billion just two years earlier. Forrester opened 2026 predicting at least two major cloud outages this year. We’re barely half done and the thermal event already spent one of them.
What to do before the next zone cooks
Force the failover, on purpose, soon, and while everyone’s still awake.
Pick a zone, put a game day on the calendar, and take that zone away from your own system while the whole team is at their desks and watching dashboards on purpose. Not in prod on the first attempt if that genuinely terrifies you, fine, do it in staging first, but staging that actually mirrors your zone topology, because a game day against an architecture that doesn’t match production is just theater with extra steps and a false sense of safety at the end of it. Then watch what breaks, because something will. A team running this drill for the first time always, without exception, turns up at least one thing that was labeled redundant and wasn’t, and finding it on a Tuesday afternoon with the whole team looking on costs you approximately nothing next to finding it at midnight because a building in Virginia ran a fever.
A few things I’d check by hand, since they’re the usual culprits: whether your databases actually promote a replica with no human in the loop, whether your other zones hold real spare capacity or just the on-paper ability to scale into capacity that may not physically exist when everyone lunges for it at once, and what your true end-to-end recovery time is when measured with a stopwatch instead of estimated in a design doc.
Most teams have never run this drill even once. If that’s you, you don’t have failover. You have a slide, and a fever waiting to happen in a building you will never see.
I broke down what silent reliability failures actually cost here.
— Youn