Microsoft traces global Microsoft 365 outage to a bug from a July update
A preliminary report says a dormant defect in gateway servers, a Teams failover and reduced capacity combined to knock out more than 80% of those servers.
Key takeaways
- Microsoft says a latent bug from a July update to its network gateway servers caused the Nov. 12 Microsoft 365 outage.
- A Teams failover added traffic while nearly 14% of gateway capacity was already offline, which set the bug off.
- At the lowest point more than 80% of the gateway servers were unresponsive, Microsoft says.
- Microsoft says its load tests used weaker hardware with fewer cores, so the defect was not caught before release.
- Exchange Online could not move to other infrastructure, so Outlook was down longest; a final report is expected this week.
Microsoft says last week's worldwide Microsoft 365 outage was caused by a software defect introduced in a July update to its network gateway servers, which stayed dormant for more than three months before heavier traffic and reduced capacity set it off. The explanation comes from a preliminary post-incident report the company shared with customers in the Microsoft 365 admin center, according to BleepingComputer and The Next Web.
The outage began on Thursday, Nov. 12, and hit Outlook, Exchange Online, Teams and Copilot around the world, BleepingComputer reports. It says the disruption lasted nearly 24 hours and that Microsoft confirmed active customer impact had ended by 11:10 a.m. ET on Nov. 13.
What went wrong?
According to the report, as described by both outlets, three things happened at once. The July update contained a latent bug that could overwhelm a server under certain traffic conditions. The Next Web reports that the flaw led to lock contention in an internal Windows process, exhausting the CPU and memory of a server.
Traffic then rose after Microsoft Teams failed over to the gateway infrastructure that also carries Exchange Online. At the same time, an unrelated problem had already taken nearly 14% of the gateway's server capacity offline, Microsoft says. BleepingComputer reports that the remaining capacity would normally have been enough, but the combination triggered the bug.
When one server stopped responding, its traffic moved to the next one, which then hit the same defect. "In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive," Microsoft wrote, adding that at the lowest point more than 80% of servers were unresponsive. BleepingComputer reports that safeguards meant to stop problems spreading between regions also failed to contain it.
Why did testing miss it?
Microsoft says its load testing mostly used less powerful machines with fewer cores, while the contention problem gets worse on newer machines with more cores, according to both outlets. That is why the defect was not caught before release, the company says.
Why did recovery take so long?
Restarted servers were quickly overwhelmed again by traffic that had built up, Microsoft says. Charles Lamanna, who leads Copilot, agents and platform as an executive vice president at Microsoft, wrote on X that because the defect came from a change made months earlier, the company's normal rollback, flighting and feature flags could not quickly undo it, The Next Web reports. According to the report, as described by The Next Web, engineers instead wrote an update that put the Windows code path back to its earlier state and rolled it out to batches of servers one after another.
Services such as Teams, OneDrive, SharePoint and the Office apps were able to shift to other infrastructure and recovered sooner, but Exchange Online could not make that switch, which left Outlook users without their mailboxes the longest, both outlets report. The Next Web says Exchange Online could not switch for reasons of scale. BleepingComputer notes that Teams also had a separate outage earlier on Nov. 12 and that it is unclear whether the two were related.
What happens next?
Microsoft says it has added capacity, improved monitoring and added safeguards so that traffic redirection cannot set off a similar chain of failures, BleepingComputer reports. Lamanna apologized for the disruption on X. BleepingComputer says a final post-incident report is expected later this week; The Next Web, citing The Register, puts the date at Nov. 20.
Andrew Garbarino, who chairs the House Homeland Security Committee, told Politico he wants more information from Microsoft and from CISA, The Next Web reports. On Nov. 13, Microsoft said the cause was not a cyberattack and was not related to AI agents.
Questions readers are asking
What caused the Microsoft 365 outage?
Microsoft says a dormant defect from a July update to its network gateway servers was triggered by extra traffic from a Teams failover while part of the server capacity was already offline.
Was it a cyberattack?
Microsoft says no. On Nov. 13 it said the outage was not caused by a cyberattack and was not related to AI agents.
Why was Outlook affected the longest?
Other services such as Teams, OneDrive and SharePoint could switch to other infrastructure, but Exchange Online could not, according to Microsoft's preliminary report.
Sources
- #microsoft
- #microsoft-365
- #outage
- #exchange-online
- #cloud-reliability
Get the day's top AI & tech stories on WhatsApp: Follow the NextFish channel
Get stories like this every morning.
Related
Policy & Safety3 min
Duffy ties FAA ground stop to contractor's copper-to-fiber work
The transportation secretary says a router flooded the air traffic network after L3Harris upgraded a remote site. Officials have not yet found the root cause.
Big Tech3 min
Italy fines Meta €805 million over Facebook's phone gallery access
The Garante says the Facebook app for Android accessed photos and videos on users' phones without specific consent. Meta has 60 days to change how the feature asks for permission.
Cybersecurity3 min
FBI reportedly examines software update after FAA ground stop
A telecom failure halted US departures for more than four hours on Friday. Officials see no sign of a cyberattack, and accounts of the cause vary in detail.
Big Tech2 min
Amazon weighs 'World Product Graph' to index nearly every product: Reuters
Reuters reports an internal Amazon plan to gather data on items sold anywhere, partly to counter AI shopping agents. Amazon says the draft may not reflect its plans.