Skip to content

Advertisement

DevOps Society
CloudAnalysis

Two Microsoft Outages in Two Weeks Show Where Single Points of Failure Hide

An Azure networking outage hit around 18 regions on 1 October, and Microsoft 365 users behind Global Secure Access lost access for almost 11 hours on 13 October. Both failures sat in plumbing most teams never think about.

Published your local timeupdated

Two Microsoft Outages in Two Weeks Show Where Single Points of Failure Hide

Neither of Microsoft's recent outages took down a headline service like Outlook or Azure VMs across the board. They hit the layers underneath, the ones that connect you to everything else. That is why they are worth a closer look.

1 October: the gateways

Overnight in Europe on 30 September into 1 October, Azure had a networking incident across roughly 18 regions. Reports differ slightly on the count and the timing, but it ran for several hours and covered North and West Europe, UK South, parts of Asia and the US.

The affected services were ExpressRoute Gateway, VPN Gateway, Azure Firewall management, Application Gateway and Azure VMware Solution. In plain terms, the private links between companies' own data centres and Azure got slow or dropped.

Microsoft's early explanation was candid. A recent change to the regional gateway management service created unexpected load during routine infrastructure maintenance, and the service could not scale to cope. Microsoft rolled the change back and promised a full incident review within 14 days.

13 October: the security gateway

Twelve days later, Microsoft 365 users who connect through Global Secure Access, Microsoft's zero trust network service, lost access to Outlook, OneDrive, SharePoint, Teams and Windows 365 Cloud PCs. Users in Germany were hit, and the UK was in the affected list for most of the day. It lasted close to 11 hours, from 07:40 to 18:31 UTC.

Microsoft said part of the GSA infrastructure was "performing below acceptable performance thresholds". The workaround it eventually suggested, around 18:20 UTC, was to switch GSA off.

That workaround says a lot. The service that was supposed to make access safer became the thing standing between ten hours of staff and their email.

What we would take from this

Most resilience planning focuses on compute and databases. Multi-AZ, replicas, failover regions. Far fewer teams have a plan for the day their hybrid connection or their security proxy fails, even though both sit in front of everything.

Ask your team three uncomfortable questions. If ExpressRoute or your site-to-site VPN to Azure went slow for six hours, which internal systems would break, and is there a backup path? If your zero trust or secure web gateway failed, could people still work, and who has the authority to turn it off? And does your status monitoring watch these layers at all, or only the apps behind them?

The second question is the one that usually gets awkward. Security teams rightly do not want a big red bypass button. But an outage with no agreed fallback ends with someone making that call under pressure at 3pm, which is worse. Decide the rules now: who can approve a temporary bypass, for how long, and what extra logging switches on while it is active.

Microsoft's incident review for 1 October should be out around now. When it lands, it is worth reading in full with your network team. Vendor post-mortems are one of the cheapest ways to find gaps in your own design.

Advertisement

Follow DevOps Society on LinkedIn

Practical infrastructure engineering in your feed.

Follow

Written by

DevOpsSociety Editorial Team

Editorial Team

The DevOpsSociety Editorial Team covers DevOps, cloud infrastructure, Kubernetes, AI infrastructure, platform engineering, cybersecurity, FinOps, and modern engineering practices. We publish practical insights, technical guides, architecture analysis, and research for engineers and technology leaders.

More from DevOpsSociety →
The Infrastructure Briefing

Get the infrastructure briefing.

Practical DevOps, cloud, AI infrastructure and engineering insights, delivered weekly. Read by engineers and engineering leaders.

No spam. Unsubscribe anytime.

Two Microsoft Outages in Two Weeks Show Where Single Points of Failure Hide, DevOps Society