The past year was filled with big IT news: The hype surrounding AI and its widespread adoption, huge numbers of tech industry layoffs, major cyberattacks, and major mergers.
Like most years, 2024 also saw its share of IT disasters. Some were forgotten within days or weeks, but others have had long-lasting effects, with damages running into the billions of dollars in one case.
For this list, we’ve largely ignored data breaches, with other lists available for those interested. And we’ve already published a list of recent AI failures, several of which happened in the past year. Fast food giant McDonald’s, for example, dumped an AI-based ordering system in June after it wouldn’t stop adding food to customer bills.
[ For past IT mishaps of note, see our biggest IT failure roundups from 2023 and 2021. ]
The mother lode of meltdowns
A faulty software update from cybersecurity vendor CrowdStrike in mid-July caused about 8.5 million computers running Windows to crash to the blue screen of death, then go into a repeating boot loop. Windows machines in endless boot loops are pretty much useless, beyond serving as door stops or paperweights.
Windows systems at hospitals, airline flight reservation centers, emergency response centers, and public transportation services were among those affected by the outage. The outage was still causing hundreds of flight cancellations and other problems 24 hours after initial reports. Some estimates put the cost of the disruption at more than $5 billion.
CrowdStrike blamed a hole in its software testing tool for the flaw in a sensor configuration update released to Windows systems on July 19. The flaw was in a type of exploit signature update known as Rapid Response Content, which goes through less rigorous testing than some other CrowdStrike updates.
The fallout after the outage was swift but continues even today. The incident prompted some CIOs to rethink their dependence on cloud infrastructure, and it caused Microsoft to focus more on kernel-level access for other software packages.
Meanwhile, Delta Airlines filed a $500 million lawsuit against CrowdStrike and Microsoft and reconsidered its use of Microsoft products.
Millions of missed calls
While the CrowdStrike outage was huge, it was dwarfed in sheer numbers by a February service interruption at AT&T Mobility, affecting 125 million mobile devices across the US. The outage, lasting more than 12 hours, prevented about 92 million calls from being completed, including 25,000 911 calls, according to the US Federal Communications Commission. An equipment configuration error was the cause of the massive outage.
While it took AT&T close to two hours to roll back the network change, restoration of full service took at least 12 hours because the mobile carrier’s device registration systems were overwhelmed with the high volume of requests for re-registration onto the network, the FCC said.
Then in June, AT&T customers reported another service outage. Reports of service outages began to spike at 1 p.m. EST on June 4, then declined around 6 p.m. The areas surrounding New York City, Chicago, Philadelphia, Dallas, Pittsburgh, and Indianapolis were apparently affected.
McOutages
McDonald’s experienced more IT problems beyond an AI ordering system that thought customers wanted 200-plus orders of Chicken McNuggets. In March, a widespread outage affecting credit card orders, including online and kiosk payments, lasted for about 12 hours.
McDonald’s restaurants in the Far East, Europe, the US, and Australia reported problems with credit card payments, with the problem blamed on the much-feared third-party configuration change. The company’s global CIO didn’t offer details but noted that the outage was not related to a cybersecurity attack.
Dangers of third-party software updates
McDonald’s wasn’t the only company plagued with point-of-sale outages. UK supermarkets Tesco and Sainsbury’s and bakery chain Greggs all experienced problems with their third-party operated POS systems about the same time that McDonald’s reported the same kinds of issues. In most cases, problems were resolved within a business day, but companies were unable to process credit card payments during that time.
In some cases, affected companies reported that the problems were related to software updates, raising questions about the reliability of third-party POS providers.
Chatbot gone rogue, part 238
You’d think people would learn a thing or two after the same thing keeps happening. Microsoft in February launched an investigation of its Copilot AI chatbot after reports on social media of it taunting users who suggested they were considering suicide. Microsoft found the bad responses were a result of a prompt injection attack, in which users are able to override safety controls in a large language model AI. The bad responses were limited to a small number of responses, according to Microsoft.
But the thing is this isn’t the first Microsoft chatbot that’s gone rogue. In early 2023, an AI chatbot packaged with Bing started professing love to some users and insulting others, calling them ugly and comparing them to Hitler.
And way back in 2016, Tay, an experimental AI chatbot Microsoft let loose on Twitter, voiced support for genocide and for Nazis. We sense a trend here.
The February investigation into Copilot wasn’t the only problem with Microsoft’s AI assistant this year. In November, Microsoft rolled out new tools to prevent Copilot from oversharing data such as employee confidential information.
Financial aid fiasco
In March, the US Department of Education said it discovered an error in the calculations of financial aid for hundreds of thousands of college students, leading to a delay in aid awards.
The department blamed a vendor working for the federal government for incorrectly calculating the financial aid formula, affecting more than 200,000 students.
The miscalculation happened at the same time as the Education Department overhauled the Free Application for Federal Student Aid, or FAFSA, which is used to determine eligibility for federal Pell Grants and other financial aid.
The vendor apparently didn’t factor some assets, such as investments, into some students’ financial need, resulting in scores showing they had more financial need than they actually did.
At the same time, the department’s overhaul of the FAFSA form created delays in the financial aid application process. Although the form is usually available to fill out in October, it wasn’t ready until late December, and the department didn’t start processing the forms and sending them to states and colleges until March. The form was available only sporadically between late December and early January.
The Education Department ran into several glitches and bugs while processing the forms. One glitch blocked parents without Social Security numbers from filling out the form.
It’s unclear how many of these problems were pure IT disasters or IT disasters mixed with user error. It seems like there were a combination of mistakes.
Friendly fire
In February, Chinese PC maker Acemagic acknowledged shipping machines with malware installed on them. Oops.
The admission came after YouTuber The Net Guy found malware on Acemagic mini PCs when testing them. The PCs shipped with the Backdoor.Bladabindi malware, as well as RedLine Stealer.
The company, in an odd explanation, blamed developers who tried to cut boot times with some software modifications.
The real terminator
The UK version of the Post Office, which is, as you’d expect, a government-run delivery service, fired more than 700 employees in late 2023 and early 2024 on the recommendation of its Horizon IT system, built by Fujitsu. It turns out that the IT system accused the former employees of stealing money from the service by falsely claiming that funds were missing from accounts they controlled.
Some news reports suggested that Horizon, installed way back in 1999, didn’t share documentation of known errors with its Post Office overseers. In addition, Post Office employees had complained for years about falsely reported missing funds.
The Post Office in 2023 had tried to move away from Horizon and migrate to the cloud, but that effort was unsuccessful while costing £31 million. In late January, Fujitsu was suspended from bidding on UK government contracts.
Read More from This Article: 8 major IT disasters of 2024
Source: News