Skip to content

ResourcesNews

Microsoft 365 outage bug slipped past tests on older servers

Microsoft's final report says a months-old Windows bug froze newer gateway servers, then its own traffic routing spread the damage to every region.

By Jason NguyenPublished 3 min read
A row of plain upright blocks on a line, with the first block tipping into the next
Image: Illustration / Recaply

In short

  1. 01Microsoft's final report says a hidden Windows bug left more than 80% of North American gateway servers unresponsive at the low point.
  2. 02The Exchange Online incident ran for 23 hours and 50 minutes, and Microsoft says it found no enterprise email permanently lost.
  3. 03Microsoft lists 13 next steps, including load tests on its newest, highest-core servers by the end of March.

The story

Microsoft just published its final report on the outage that kept most Exchange Online users out of their mailboxes. The incident ran for 23 hours and 50 minutes, and its cause, a Windows bug, had slipped past load tests that ran on older servers. Microsoft shares the report in its admin center, so our link goes to a public copy.

The bug came from an update used only on Microsoft 365's network gateway servers, the front door for Outlook and other apps. GeekWire's read of the report also caught a change in Microsoft's account. The preliminary version blamed an unrelated issue for missing server capacity, while the final one calls it routine deployment work.

A routine failover and peak traffic set off the hidden bug

The defect caused lock contention inside a Windows process until servers ran out of CPU and memory. It hit only newer hardware, because more cores meant more concurrent requests. Microsoft's lab tests used older machines and a different traffic mix, so they missed it.

The update ran for more than three months without trouble. Then three routine conditions lined up.

  • Nearly 14% of gateway capacity was offline for a routine service deployment.
  • A routing issue in US East moved all Teams traffic worldwide onto the same gateways about 15 hours earlier.
  • North American traffic climbed toward its daily peak.

Microsoft's own traffic systems spread the failure worldwide

As servers froze, automatic routing moved their traffic to healthy servers that carried the same bug. More than 80% of North American gateway servers went unresponsive at the low point. Within about two hours of those broad failures, servers in every region were affected.

Recovery stalled for a similar reason. Servers coming back were swamped by waiting clients and froze again, and normal rollback tools couldn't quickly undo a months-old change. Each server needed a restart, a patch and a second restart.

Of the top 30 comments in the Hacker News thread on the preliminary report, seven brought up AI as a possible factor. That's a snapshot, not a survey. Microsoft's final report says the change "was not caused by AI tooling."

Some of Microsoft's 13 fixes still have no date

By the end of the year, Microsoft plans to stop failures from cascading between regions and to fail Teams over only in the affected region. Load tests on its newest, highest-core hardware are due by the end of March. Gradual traffic ramp-up for recovering servers has no date yet.

Microsoft says it found no enterprise email permanently lost, though some delayed messages to consumer mailboxes inactive for over 30 days bounced. Admins can read the report in the Microsoft 365 admin center under EX1550543 and MO1550821. The public copy linked above comes from 365Fanatic.

Why it matters

Microsoft's safeguards did their jobs and still made things worse, because every server they moved traffic to had the same bug. A failover drill at off-peak load on older machines won't catch that. Test on the hardware you actually run, at peak, and assume a shared defect can follow your traffic anywhere.

Get Recaply DevOps in your inboxFree, unsubscribe any time.

More stories