Why Delta was hit so hard by the global IT outage

He also explained that the airline still has not restored full functionality to its operation.

About 60% Delta’s most critical systems to run the airline are Windows-based and were rendered inoperable early Friday morning, according to the airline. The problem came from a faulty security update from a company called CrowdStrike.

The CrowdStrike error “required Delta’s IT teams to manually repair and reboot each of the affected systems, with additional time then needed for applications to synchronize and start communicating with each other,” Delta said.

Bastian wrote that “one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown.”

“Our teams have been working around the clock to recover and restore full functionality,” according to Bastian.

The system to get flight crews in place in time for flights “is deeply complex and is requiring the most time and manual support to synchronize,” Delta said in a statement Monday.

Often when an airline’s operations are disrupted by severe weather, the worst of the impacts are generally confined to one region or country. But here, Delta is encountering problems that have clipped the wings of its entire network.

“It’s going to take another couple of days before we’re in a position to say… the worst is clearly behind us,” Bastian said in a video message to employees Monday. He added that “there could not have been a worse time for this to happen,” on a Friday of its busiest-ever summer weekend.

Delta has struggled to get enough crew members to staff its flights, leading in many cases to delay after delay before a flight is ultimately canceled.

In some cases, the airline may get one or two crew members to the gate for a flight, only to have them wait hours for additional crew members and “time out,” or reach the limit for hours on duty.

When that happens and the flight is canceled, Delta loses the use of those crews, while the aircraft and crews don’t make it to the next city for their other flights for the day, and the problem snowballs — across hundreds of flights around the country and the world.

Credit: Ziyu Julian Zhu/AJC

Credit: Ziyu Julian Zhu/AJC

“It’s been a horrible weekend,” Bastian said to employees Monday. “This issue came out of nowhere.”

According to Delta Chief Information Officer Rahul Samant, Delta got most of its Windows-based applications back up and running to operate its first flight in Atlanta at around 7:30 a.m. Friday morning. But an application that manages arrivals and departures at gates in Atlanta uses so much data that it took a couple more hours to get back up and running, slowing the restart at its largest hub.

And the crew tracker application for flight attendant and pilot assignments had more than 2,000 items that needed to be fixed in its queue, according to Samant. On Friday and Saturday the company built more versions of that system to try to catch up on the backlog. “We’re working on, how do we resync, how do we reset?” Samant said in the video message to employees.

Under different circumstances, Southwest Airlines in December 2022 had a meltdown of its operations that started amid severe winter weather during the busy Christmas holiday travel period and worsened as days wore on due to problems with its crew scheduling software.

Passenger Shakim Tilley, who was at the airport for about 12 hours Friday with his wife and children through multiple delays before his flight was canceled, said Delta told passengers the delays were because they didn’t have the flight attendants needed to operate the flight.

“They kept delaying us because they said we didn’t have a flight crew,” Tilley said.

“I was trying to recruit flight attendants,” said Tilley’s wife Tiffany.

“The pilot sat on the airplane for 10 hours. He timed out,” Shakim Tilley said. “Finally we got our flight crew,” but because the pilot had timed out, “then they canceled the flight.”

While some companies that have to shut down operations for several hours can resume operations relatively smoothly afterward while catching up on a backlog of work, it can be much more complex for airlines.

Flight schedules, crew schedules and passengers’ connecting itineraries amount to an intricate and carefully-timed choreographed ballet of interconnected operations, and a disruption can leave all of the pieces of out of place. That makes it much more complicated to resume operations cleanly.

That means such disruptions pose huge tests for airlines’ recovery operations — which include crew scheduling and tracking.

An airline like Delta with operations in hundreds of cities around the world needs to track where crews are available to be able to staff flights, which becomes much more complicated when flight schedules are disrupted and crews are displaced.

With the outage having lasting effects on its crew tracking capabilities, Delta failed to adequately recover quickly after the outage and has struggled to return to normal operations in the days since.

Credit: arvin.temkar@ajc.com

Credit: arvin.temkar@ajc.com

Delta passenger David Fields experienced this firsthand and ended up stuck in Atlanta for days after his connecting flight was canceled and he tried to fly standby for multiple flights.

“Delta just kept canceling... flight after flight,” Fields said Saturday. Flights changed gates multiple times, forcing all of the passengers waiting in the gate area to move between concourses through the night, he said.

“We just ended up going all night, back and forth, for hours and hours and hours,” Fields said.

Well after midnight when Plane Train service is cut for its overnight operations, passengers including those pushing elderly people in wheelchairs had to walk between concourses, he said. Ultimately, without being able to get on any flight, Fields was rebooked for a flight three days later, and booked a hotel in Midtown Atlanta.