Your service ran in two zones for exactly this reason. Then, on 1 September 2026, both zones lost the network together, and the cause had nothing to do with power, weather or software. A maintenance procedure that should have touched one router at a time touched all of them in one sitting. Google Cloud's us-central1 outage is a useful case study because the technology held up fine. The change process didn't, and your change process has the same weak spots.
Here's what Google says happened, why one procedural slip crossed a zone boundary, and what to change in your own change management this week.
What happened in us-central1
According to Google's incident report, the disruption ran from 07:41 to 11:52 US/Pacific on 1 September, a little over four hours. It hit two zones in the Iowa region, us-central1-b and us-central1-f. Google lists 15 affected products, and the list reads like a production stack: Compute Engine, GKE, Cloud SQL, Spanner, AlloyDB, Bigtable, BigQuery, Dataflow, Cloud Run, App Engine, Filestore, Apigee, Looker, VPC and Hybrid Connectivity.
The trigger was physical. Technicians were replacing optical transceivers on data-center routers as part of a scheduled capacity upgrade, and the plan was to do one router at a time over several days. A procedural error meant the technician received the full list of transceiver replacements across every router, with no instruction to sequence the work. So they worked through the list. Within 13 minutes, every fiber path across the affected devices had been unplugged.
Two more things went wrong. A standing rule said to stop if light was still detected on a fiber after unplugging it, which is the signal that the link is live and carrying traffic. That rule wasn't followed. And the maintenance workflow lacked the human and software verification steps that should have caught an unintended disruption while it was happening.
Google's automation then took roughly 19 minutes to move regional traffic away from the damaged zones. The remediation list is specific: finish moving network upgrade workflows onto a fully sequenced, automated orchestration system; complete work-stop alerting for any action that disconnects a live fiber; and cut the traffic-shift time to around five minutes.
That write-up deserves credit. Google named the procedural error, admitted a safety rule was skipped, and put a number on the detection gap. That candour is what makes the incident worth studying.
One work order reached two zones
Multi-zone design rests on an assumption you rarely write down: whatever breaks zone B won't break zone F at the same moment. Power, cooling and physical separation mostly deliver that. But zones are also joined by things that aren't physical: shared control planes, shared config pipelines, shared on-call rotations, and shared maintenance plans.
This time the shared thing was a work order. One list, handed to one person, covering devices that served both zones. Once that list existed without sequencing, the zones had a common failure mode, and it was the change process.
Your stack has the same coupling, probably in more places than you'd guess. A Terraform apply that targets every zone's node pool in one plan. A Helm upgrade that rolls the DaemonSet across the whole cluster. An Ansible playbook with serial unset. A DNS or firewall change pushed globally because the tool makes global the default. Each of these turns "we run in three zones" into "we run in three zones until someone changes something". The AWS architecture guide on failure domains and multi-account design covers laying out the physical side properly. This incident is about the other half: the humans and pipelines that reach into every zone at once.
My position: if one change can touch more than one zone without a gate between them, you don't have zonal independence during that change. Plan your capacity and your SLOs as if you don't.
A safety check that relies on memory will eventually be skipped
"Stop if you see light on the fiber" is a good rule. It's cheap, it directly detects the dangerous condition, and it was already written down. It still failed, because it depended on a person remembering to do it, under time pressure, while working through a list that looked routine.
That's the gap between a runbook step and an interlock. A runbook step says "check X before doing Y". An interlock makes Y impossible, or at least loud, until X is true. Google's own remediation moves in that direction: work-stop alerting fires when a live fiber is disconnected, whether or not anyone remembered to check.
Most teams have runbook steps pretending to be controls: "confirm the replica is caught up before promoting", "drain the node before patching". If a tired engineer can skip it, it's advice, not a control.
Blast radius was set by change scope
The plan was one router at a time. That sequencing was the real safety mechanism, more than any redundancy in the hardware. One router out of service is a capacity dent the network absorbs. All of them out in 13 minutes is an outage. The hardware was the same in both cases, and only the batch size differed.
This is the most transferable lesson. For most production changes, how much you change before you check is the variable that decides whether a mistake is an incident or a non-event. And batch size is often decided by accident: by what fits in a ticket, by whatever the tool defaults to, or by a list that got handed over whole.
Step in the failure chain | What should have stopped it | Why it didn't |
|---|---|---|
Work order issued with every router on it | Sequencing built into the plan, one device per step | Procedural error; no instruction to sequence |
First fiber pulled while still carrying light | Standing rule to halt if light is detected | Relied on a human; not followed |
Further paths pulled on other routers | Verification between devices (human and software) | Those steps were missing from the workflow |
Both zones lose network | Automated detection and traffic shift | Worked, but took about 19 minutes |
Read down the middle column. Every control existed as intent. None of them was built into the thing doing the work.
What to change in your own change management on Monday
You can't change how Google sequences router maintenance. You can change how your own infrastructure changes move across zones. Five changes, roughly in order of payoff:
Make zone the unit of rollout. Any change touching infrastructure in more than one zone should go zone by zone, with a soak period and a health check between each. That covers node pools, network policy, load balancer config, OS patching and database parameter groups. If your tooling can't express that, fix the tooling before the next big change window.
Turn runbook checks into pipeline gates. Take your three most-used runbooks and find every "confirm", "verify" or "check". For each one, ask whether the pipeline could evaluate it and refuse to continue. Most of them can.
Make the unsafe action hard, not just discouraged. Default
serialto 1 in Ansible. SetmaxUnavailableon DaemonSets and a PodDisruptionBudget on every stateful workload. Require an explicit, reviewed flag to target all zones at once, and alert whenever it's used.Watch for unintended impact during the change, not after it. Google's fix here is work-stop alerting. Your equivalent is an automatic halt when error rate or latency in the zone you just touched moves past a threshold, wired to stop the rollout rather than page someone who then has to stop it by hand.
Test that zonal actually means independent. Once a quarter, pick a change your team makes routinely and trace every zone it touches. Then run a game day where one zone is degraded mid-change and see whether the pipeline halts or keeps going.
A rollout policy that encodes the first four points doesn't need to be clever. Something like this, expressed in whatever your deployment tool supports:
# Change policy for any infrastructure change spanning zones
rollout:
unit: zone # never batch across zones
order: [us-central1-a, us-central1-b, us-central1-f]
soak_minutes: 30 # time for slow failures to surface
pre_checks:
- peer_zones_healthy: true # refuse to start if another zone is already degraded
halt_on:
- zone_error_rate_increase_pct: 2
- zone_p99_latency_increase_pct: 25
all_zones_override:
allowed: false # flipping this needs a second approver and fires an alert
The peer_zones_healthy pre-check is the one teams skip, and it's the one that matters here. Taking one zone down on purpose is fine only if the others are fully up. Starting a zonal change while a peer zone is already struggling is how a planned action lines up with an unplanned one.
Another outage from the same month
The us-central1 outage wasn't September's only big cloud incident. On 16 September 2026, Salesforce had a disruption lasting roughly three and a half hours that affected logins and APIs, in the middle of its Dreamforce conference. Salesforce has not published a full root-cause analysis, so there's nothing to learn about its change process yet, and we won't guess. More incident analysis lives in the DevOpsSociety cloud section.
A decision rule to take away
Before approving any change, ask one question: if the person or pipeline making this change makes a mistake at step one, how many zones can it reach before something forces a stop? If the answer is "all of them", the change isn't ready, however good the runbook is. Google's own fix is to sequence the work automatically and alert on the unsafe action. Yours should be too.
