Microsoft’s 30-Hour Outage Priced the Cost of Losing Its Legacy Engineers

Microsoft’s servers suffered a global outage that ran for 30 hours. Outlook mail and Teams meetings both failed, and hours after the company declared recovery, messages in some parts of the world still had not arrived.

The apology volunteered that the incident had nothing to do with AI, before anyone had asked. Copilot went down as well, because it could not read the mail it depends on.

Satya Nadella speaking beside a graphic of Microsoft Outlook, Teams, Copilot and SharePoint application icons
Copilot stopped working because it could not read the underlying mail system (Source: Sohu Tech)

A bug, then a slow repair

The trigger was ordinary. Engineers adjusting traffic flow during routine network maintenance activated a latent bug, and servers began dropping in batches. Google and Amazon have had similar incidents. No cloud provider can guarantee clean code.

The unusual part was the repair. Reading Microsoft’s dozens of incident updates in order, the first hours were spent guessing. The company said traffic had not risen and servers had failed on their own, then said nothing had changed recently. Late in the night it tried rolling back a network change that happened to match the timeline. Nothing improved.

On Blind, a verified Microsoft employee offered a comparison. A rollback that once took 45 minutes took five hours this time.

Why nobody knew the system any more

That employee argued the cause is two years of management attention concentrated on shipping AI features. Work maintaining old systems, the argument goes, brought no promotion even when done well, and raises below 5 per cent. After several rounds of layoffs and resignations, almost everyone who understood how the underlying routes connected had left. At 3 a.m. on the day of the outage, the people still on the call faced a black box. The engineers who wrote the code had departed two years earlier.

Microsoft’s own personnel moves point in the same direction. In late October it offered a voluntary departure package to 27 per cent of staff, roughly 60,000 people, effective at the end of the year. Voluntary schemes tend to be taken first by those who can find another job easily, and engineers who have kept a mail system running for a decade are in demand.

Copilot makes the irony sharp. Microsoft has spent two years building it up and funded much of that push with savings from layoffs. Yet Copilot needs to read a user’s mail, calendar and files, all of which sit in the old system. When that system stopped, the AI was blind.

The deeper concern is dependency. As more systems are written by AI and become unreadable to people, repair may rely on another AI running on the same cloud. When the cloud fails, the fixer and the broken thing fail together.

Microsoft said the incident was unrelated to AI. What readers asked was who would fix the next one. The most-liked reply to the apology requested that a human write the incident reports.

A business owner in the comments said he lost USD 250,000 in a day and had already asked IT to find a backup mail provider. Large customers may start writing mandatory human staffing for critical roles into cloud contracts, and the engineers who know the old systems may return as consultants at consultant rates.

Companies have calculated the savings from replacing staff with AI for two years. Microsoft put the price of one outage day on public display, and handed the bill to customers.

Editor’s note: This is an adapted translation of the original Sohu Tech report. It has been trimmed and restructured for readability for an international business audience. The full original (in Chinese) is at https://www.sohu.com/a/1102502252_473283.

Leave a comment