Outages Don't Break Systems. Weak Assumptions Do.
On 18 November 2025 a Cloudflare outage took down access to X, ChatGPT and a long list of other sites. A month earlier, on 20 October, an AWS incident in us-east-1 rippled into thousands of downstream services, and on 29 October Azure Front Door failed globally for about eight and a half hours.
When Cloudflare or AWS goes down, nothing you run fixes it. You post a status update, watch the provider's incident page, and bring your own services back in order once theirs return. That part is unavoidable.
The real risk usually sits inside the walls, and it's almost always self-inflicted. External outages expose systems. Internal weaknesses take them down.
What the three postmortems actually say
All three providers published their own accounts, and they're worth reading for what they teach about systems you do control. Each was caused by a change inside the provider, not by an attack, and each took far longer to recover from than to trigger.
Cloudflare, 18 November 2025. A permissions change to a ClickHouse database cluster at 11:05 UTC made the query that builds the Bot Management feature file return duplicate rows. The file roughly doubled in size. The proxy reading it had a hard limit of 200 features, "well above our current use of ~60 features", and the oversized file broke through it, so the proxy failed and returned 5xx errors. Core traffic was largely restored by 14:30 and everything was back by 17:06 (Cloudflare's postmortem). The lesson for everyone else: a generated configuration file is code, and it needs the same validation before it ships.
AWS, 19 and 20 October 2025. A latent race condition between two automated DNS Enactors left DynamoDB's regional endpoint in us-east-1 with an empty DNS record, starting at 11:48 PM PDT. DNS was fixed by 2:25 AM. EC2 instance launches didn't fully recover until 1:50 PM, because the system that manages EC2 hosts had fallen into congestive collapse while DynamoDB was unavailable, and Network Load Balancer health checks then misfired until engineers disabled automatic failover (AWS's summary). The trigger lasted under three hours, and the recovery took most of a working day. Any design that needs to launch new capacity in order to recover was waiting on the slowest part of that timeline.
Azure Front Door, 29 October 2025. From 15:41 UTC on 29 October to 00:05 UTC on 30 October, Microsoft's global edge and CDN service failed, taking dependent services such as App Service, Entra ID and Azure SQL Database with it. Microsoft's post-incident review (tracking ID YKYN-BWZ) traces it to valid customer configuration changes, made across two control-plane versions, that produced metadata the data plane couldn't process. The crash came about five minutes after deployment, too late for the protection system to catch, so the bad configuration spread globally. If your only route to your application runs through one edge provider, its outage is your outage.
Where teams actually get hurt
Most serious incidents don't start with a cloud provider. They start with unverified assumptions.
- A backup exists, but no one has restored it under real conditions.
- A recovery path works in a test environment but collapses at production scale.
- Credentials haven't been rotated because nothing bad has happened yet.
- Monitoring fires alerts but doesn't tell anyone what actually broke, or who owns fixing it.
None of this feels urgent in isolation. Together, it turns a routine upstream outage into a prolonged business incident.
We've seen this pattern repeatedly when initially engaging with enterprise clients. The primary database restore has never been tested at production scale. The backup exists, the restore path fails, and what should be a short disruption stretches into hours or days while teams debug something they assumed worked.
By the time the cloud provider has fully recovered, the internal outage is just getting started.
Prove the restore, with a stopwatch
"We have backups" is a statement about storage. The number that matters is how long a full restore takes, and nobody knows it until they've done one. Two ways to find out this week.
On Amazon RDS, restore production to a new instance from the latest restorable time. The restore command returns as soon as the request is accepted, so time the wait for the instance to become available, which is the part that takes the time:
aws rds restore-db-instance-to-point-in-time \
--source-db-instance-identifier prod-orders \
--target-db-instance-identifier restore-drill-2026-10 \
--use-latest-restorable-time
time aws rds wait db-instance-available \
--db-instance-identifier restore-drill-2026-10
The waiter polls every 30 seconds and gives up after 60 checks, so a restore longer than half an hour needs the wait rerun until it succeeds. Add those times together.
On self-managed PostgreSQL, restore last night's dump into an empty database with parallel jobs:
createdb restore_drill
time pg_restore --jobs 8 --dbname restore_drill /backups/orders-latest.dump
Then check it. Compare row counts on your five largest tables against production, run the three queries your support team uses most, and point a staging copy of the application at it. Write down the elapsed time next to your recovery time objective. If the restore takes six hours and the objective is one, you've learned that on a quiet Tuesday rather than during an incident.
A quick reality check for leaders
If any of the questions below make you uncomfortable, you're not alone, but they're worth asking.
- When was the last time your team restored a full production backup end to end, and how long did it take?
- Have you ever watched your system degrade under partial failure, or only seen it fully up or fully down?
- Do database writes consistently succeed or fail as a unit, or can partial state leak through?
- During an incident, is it obvious who is in charge, what decisions they can make, and what success looks like?
- If a major dependency failed tonight, would your system pause cleanly, or keep half-working and corrupting itself?
Most teams can't confidently answer these. Usually nobody is careless. Nobody ever forced the system to prove it.
Hardening what you actually control
Reliable systems rehearse failure. Teams that recover quickly do the unglamorous work early.
- Restore backups on a schedule, and record the time each restore took.
- Design for static stability. AWS's own Builders' Library article on static stability argues for running with enough capacity already provisioned that recovery doesn't depend on launching more, the dependency that held back EC2 launches in us-east-1 until the afternoon of 20 October.
- Treat generated configuration as code. Validate size, schema and value ranges before a file reaches production, the check Cloudflare's feature file needed.
- Make side effects safe to repeat. Idempotency keys on payments, orders and emails mean a retry after an outage doesn't charge a customer twice.
- Rotate credentials on a schedule, not after a scare.
- Keep a documented way around the edge. Know how you'd serve traffic if your CDN or front door provider were down, and what that would cost in protection.
- Establish ownership before something breaks, and practise incident response while the stakes are low.
None of this prevents external outages. That isn't the goal.
The goal is simple. When something outside your control fails, your system doesn't make it worse.
The leadership difference
Outages create uncertainty. What happens next depends on leadership.
Strong leaders assume outages will happen. They're explicit about which risks matter, which don't, and how the organization responds when something breaks upstream. They communicate early and clearly about what's known, what isn't, and what happens next. That clarity keeps teams focused and prevents panic-driven decisions that cause more damage than the outage itself.
The teams that hold up under pressure aren't the ones who tried to design failure out of the system. They're the ones who made sure failure stopped where it started.
What to do if this hit close to home
If this article made you uneasy, you're asking the right question:
"Would our system fail cleanly, or cascade?"
Start with the restore drill above. It takes an afternoon and answers the question most teams are least sure of. After that, the useful work is a focused, senior-level review of your real failure paths: what breaks first, what breaks next, what should stop but doesn't, and where assumptions are quietly doing the most damage.
That's the work we do with companies when the cost of being wrong is high.