The smallest change that broke the most

When something enormous breaks, like every flight over Britain or eight and a half million computers on the same morning, it is natural to assume the cause was enormous too. In most of the big tech outages of the last thirty years, the software and network failures that stopped flights, phones and payments, it was the opposite: three letters in a flight plan, one number too many in a file, a line deleted while someone tidied up.

We read the official reports on twenty-four of them. Three are told slowly below, from the people they hit, through what the system was and what went wrong inside it, to how far the damage spread, and after that come the eight whose exact trigger has been published, set out at their real length.

On the morning of the August bank holiday, a parent waiting at Gatwick with a three-year-old found out that something had gone wrong by overhearing other passengers talking about delays, and the family got seats on a flight three days later. Somewhere else that week a woman stranded abroad with her child was handed a leaflet by her airline suggesting they make their own way home, and the flights she could find left four or five days later and cost her £900.

To see how both of them ended up there, you need to know what happens to a plane before it reaches British air. Every flight that passes through files a flight plan, which is its route written out as a list of waypoints, points on the map with short codes, and a computer at NATS, the company that runs air traffic control, reads each plan, works out where the plane will enter and leave British airspace, and hands that to the controllers.

It does this for about 800 plans an hour on a normal day, and before that morning it had handled more than 15 million of them without trouble. Then a plan arrived for a flight from Los Angeles to Paris whose route over North America passed a waypoint with this code.

Waypoint codes are not unique around the world. DVL means Devils Lake in North Dakota, which is where this plane was going, and it also means Deauville on the coast of Normandy, about 6,700 km away, a few minutes' flying from where the plane would leave British airspace on its way to Paris.

The software was supposed to ignore repeated waypoints outside Britain, and the investigators found that this rule had been written into the specification but that "the logic was missed during the coding process". So the computer took the wrong DVL, built a route that made no sense, and at 08:32 did what it was designed to do when it cannot trust its own answer: it stopped, rather than hand controllers something that might be wrong.

A backup computer was there for exactly this moment. It took over, read the same flight plan with the same code, and stopped about twenty seconds later. With both down, controllers had to type flight plans in by hand at about 60 an hour instead of 800, and to keep the sky safe they let far fewer planes in, so airlines started cancelling.

The fix itself was slow. The on-call engineer took an hour and a half to reach the site, the software's supplier was not asked for help for more than four hours, and automatic processing only came back at 14:27, about six hours after it stopped. By then planes and crews were in the wrong cities, and it took days to put them back. The board below is the result, the passengers on cancelled flights by airline, as the Civil Aviation Authority counted them afterwards.

Counting the delays as well as the cancellations, the regulator put the number of people whose journeys were hit at more than 700,000, one for every pixel below, with the 300,000 whose flights were cancelled in black.

Eleven months later Cassandra Andrews was in the airport in Atlanta, trying to get to her mother's funeral, while Delta cancelled one flight after another. "They kept canceling, and I had to go back at two o'clock in the morning and get in the line for three hours," she told Marketplace, and at one point "we started breaking down crying, because I didn't think I would get out for my mother's funeral."

Delta's computers, like those of most big companies, ran a security program called CrowdStrike Falcon, which watches a machine for signs of an attack. To see everything, it runs inside the core of Windows, the part of the system that is not allowed to fail, and to keep up with new attacks it downloads small instruction files several times a day, which go out to millions of machines at once without anyone approving each one.

At 04:09 UTC that Friday, one of those files went out with a new rule for spotting a kind of attack. The rule asked the program to check 21 pieces of information about what the machine was doing, but the program only ever collected 20, so when it went to read the 21st it reached past the end of its list into memory that held nothing it could use.

An ordinary app that does this crashes, and you open it again. Because this program was running inside Windows itself, Windows stopped the whole computer to protect it, showed the screen below and restarted, and on the way back up it loaded the same file and crashed again.

CrowdStrike noticed and pulled the file at 05:27, 78 minutes after it went out, which on the scale of a day is the thin blue stripe below. Every Windows machine that was switched on and online during that stripe had already taken it.

By Microsoft's count that was 8.5 million computers. Each square below stands for 10,000 of them.

None of them could be fixed from a distance, because none of them stayed up long enough to receive a fix. Someone had to sit at each machine, start it in a safe mode and delete the file, and at many companies the keys that unlock encrypted drives were kept on servers that had crashed as well. "Our BitLocker keys are stored inside of a service that we can't login to because our AD is down," one administrator wrote on Reddit that morning.

Airlines, hospitals, banks and broadcasters went down at the same time. Delta cancelled about 7,000 flights over five days and told investors the outage had cost it about $380 million in lost revenue and $170 million in extra costs, Mass General Brigham in Boston called off every non-urgent surgery and appointment it had booked for that Friday, and CrowdStrike reported that about 99% of the machines were back ten days later.

Rogers carries the phones and home internet of more than 12 million customers in Canada. Like every network, it relies on routers, which keep a list of routes, the directions for where to send each piece of traffic, and the full list for the whole internet is enormous, so the routers at the heart of Rogers' network were protected by a filter that kept most of it out.

That morning Rogers was partway through a seven-step upgrade of its core network. Its own system had rated this step as low risk, so it was not tested in a lab first, and at 4:43 a.m. Eastern time, while cleaning up the configuration files on the routers, staff deleted that filter. The full list of routes poured into the core routers, which could not hold it and stopped working, and every call, text and connection that passed through them stopped with them.

Nobody outside Rogers can see inside its routers, but the Internet Outage Detection and Analysis project at Georgia Tech can see the network go dark. It sends a probe to blocks of internet addresses every ten minutes and counts how many answer, and the line below is that count for Rogers across those two days.

Part of the reason it took so long is that Rogers' own staff were on Rogers phones. According to the company's filing with the regulator, its people "started swapping out our Rogers SIM cards with our alternate carrier SIM Cards" just to talk to each other, and it had to send other networks' SIM cards out to its remote sites.

More than 12 million customers lost service for most of a day, calls to 911 could not be completed, and Rogers later credited every customer with five days of service. The change itself has never been published, because Rogers filed the details with the regulator in confidence.

It happened twice more. Sixteen months later in Australia, about 90 Optus routers each switched themselves off when the number of routes passing through them crossed a limit the manufacturer had set by default to stop them being overloaded, and around 10 million customers lost service for most of a day. The regulator found that 2,145 people could not get through to 000, Australia's 911, and that Optus failed to make 369 welfare checks it owed them, and that afternoon Optus posted its advice.

In February 2024 an AT&T employee added a new piece of equipment to the network during a routine overnight window, and the Federal Communications Commission found that "no 911 calls from AT&T Mobility-served wireless devices could be routed" while voice was down, which came to more than 25,000 failed calls. Its report says the peer review that should have stopped the change "did not take place". Neither Optus nor AT&T has published the setting it changed.

For most big outages the line itself is never shown. The company describes in words what went wrong, a deleted filter or one wrong input, and leaves the exact change out of its report or hands it to a regulator in confidence. The least has been published about the outages that hurt people most directly: Rogers, Optus and AT&T, whose networks stopped emergency calls, have between them never shown the change that did it.

The backups that were meant to catch these mistakes mostly made the same one a moment later, because a backup usually runs the same software as the system it covers. Ariane 5 carried two identical guidance computers so that one could take over if the other failed, and the inquiry found that the spare had already stopped, for the same reason, one data cycle before the main one. The NATS backup read the same flight plan and failed twenty seconds after the primary.

At GitLab the live incident document admitted that "out of five backup/replication techniques deployed none are working reliably or set up in the first place", and at the FAA the damaged file turned up in the backup database too, according to reports at the time. A copy only protects you from failures that cannot reach the copy, and a small change that looks valid gets copied along with everything else.

The other reason is speed. A change that goes to every machine at once, like CrowdStrike's file, reaches all of them before anyone can see what it does. Atlassian's clean-up script deleted 883 customer sites in 23 minutes in April 2022, some of which took fourteen days to bring back, and AT&T's 1990 update had already been loaded into all 114 of its long-distance switches weeks before the afternoon it went wrong.

Most of the people who pressed enter on these days are anonymous in the reports, which is fair, and only one of them later wrote about it in the first person under their own name. Azer Koçulu pulled left-pad and 272 other packages from npm in 2016 after the registry handed one of his package names to a company. In 2024 he wrote that he remembered "leaving my desk for a few minutes", that within ten minutes a friend told him he was on Twitter, and that several months later he quit his job and left the US.

Cloudflare's chief technology officer was in a meeting in 2019 when, as he wrote afterwards, "some of these alerts hit my watch and I jumped out of the meeting I was in", and a few minutes later a colleague told him they had lost 80% of their traffic.

The GitLab engineer who deleted the database in January 2017 was, in The Register's words, "a tired sysadmin, working late at night in the Netherlands", trying to get a copy of the database to rebuild, and GitLab's own timeline of that night gives the moment in one line.

How we did this

We read the official reports on twenty-four outages where the company, a regulator or an official inquiry described the trigger in writing, listed below, and show the exact text for the eight where it has been published. Character counts are of that text as shown, leaving out line breaks and indentation. Where a number comes only from press reports we say so. The Rogers line is IODA's active-probing signal for AS812 in ten-minute steps, converted to Eastern time, and the data we pulled ends at 8 a.m. on 9 July, when about 80% of blocks were answering. The departures board is not a real board from that morning: it shows the CAA's count of passengers on cancelled flights, by airline.

Also: CAA CAP2981 interim review (passenger cases, paras 3.16 and 3.17); CBC Hamilton, 12 July 2022 (the Eby family); IODA, Georgia Tech; ACMA, November 2024; Bean review of the Optus outage; Microsoft, 20 July 2024; Delta, SEC filing; The Register, 1 February 2017; Azer Koçulu, 8 June 2024; left-pad 0.0.3 from the npm registry.