Skip to content
Theme
OnDevice Tools

GadgetsLaptops & PCs

Latent bug left Outlook down for nearly 24 hours, says report

The Verge details how a dormant networking bug, a Teams failover and reduced capacity combined to knock out Exchange Online for millions, in an incident Microsoft rated as its highest severity.

Illustration of an Outlook error message on a laptop while Microsoft Exchange servers fail in the background

Key takeaways

  • A latent bug dating to July took down Exchange Online, leaving Outlook users without email for nearly 24 hours.
  • More than 80 percent of Exchange servers became unresponsive after failover systems spread the problem worldwide.
  • The bug surfaced only when a Microsoft Teams failover and a loss of about 14 percent of gateway capacity combined.
  • Microsoft says lab testing missed the bug because it used weaker hardware with fewer cores, and a full post-incident review is due this week.
In this article
  1. How the Outlook outage unfolded
  2. The bug behind the failure
  3. Why testing missed it
  4. A cascade of three problems
  5. What happens next

How the Outlook outage unfolded

Illustration: outage, Microsoft logo, bug, server and the labels Exchange Online, Microsoft, Microsoft 365

One of Microsoft's worst service failures in years began quietly on a recent Thursday, according to The Verge. Barely an hour after staff arrived at the company's campus, Exchange Online servers, which power Outlook mail for millions of users, started to fall offline. The problem first appeared in Switzerland, then spread to Mexico and beyond as Microsoft's failover systems shifted traffic elsewhere.

Businesses, Microsoft employees and many governments could not reach their email for close to 24 hours. Sources told The Verge that Microsoft classified the incident as sev0 in its internal tracking system, the highest level, indicating a catastrophic service disruption.

The bug behind the failure

As engineers analysed crash reports, trace logs and kernel dumps, they found that some servers had entered maintenance mode under unusually high CPU loads. When engineers tried to bring them back online, waiting traffic overwhelmed them almost immediately.

Exchange servers boot their operating system over a network using a process called Preboot Execution Environment, which lets Microsoft manage updates remotely. The Verge reports that the sudden burst of traffic from rebooting servers overwhelmed the systems monitoring them, creating a reboot loop.

Crash reports and logs ruled out a cyberattack, though engineers had investigated potential attack vectors and consulted DDoS specialists while speculation spread online. The real cause was a latent bug on Microsoft 365 networking gateway servers, sitting undetected since July. Microsoft says it does not affect regular versions of Windows, because it stemmed from a fix prepared only for those gateway servers.

Why testing missed it

Charles Lamanna, who leads Microsoft's Copilot, Agents, and Platform team, described the problem in a post on X as a latent defect that caused servers to become overwhelmed under certain load conditions. His teams took over Office, Outlook and Microsoft Teams reporting after Ryan Roslansky, the previous head of Office and Teams, left the company last month.

Microsoft's preliminary post-incident report attributes the fault to lock contention inside an internal Windows process on select Windows servers. Lamanna admitted the bug went unnoticed because lab testing focused on weaker hardware with fewer cores, while the contention issue is worse on machines with more cores and newer hardware.

A cascade of three problems

The dormant bug became visible only when two other events raised the load and cut capacity. That changed the gateway's traffic patterns and increased its load.

At the same time, around 14 percent of the gateway's server capacity was out of service because of an unrelated issue from the previous day. Together, the bug, the Teams failover and the reduced capacity triggered the failure that had lain dormant for months. Recovery took longer than usual because returning servers were immediately swamped by waiting traffic, and normal rollback and feature-flag processes could not revert a change made months earlier.

What happens next

The scale of the response shows how seriously Microsoft treated the incident. Engineers held a running bridge call throughout, and at one point it reached the maximum of 1,000 participants. The Verge was told that overlapping interventions from different teams sometimes complicated coordination.

Microsoft says a full post-incident review will be published later this week. The company is expected to invest more in its testing processes so that a similar latent defect is caught before it can cause such a widespread failure. Outages happen daily, but this kind of cascading collapse is a rare opportunity for Microsoft to learn and prevent a repeat.

Sources

  1. Inside Microsoft’s giant Outlook outageThe Verge

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.