An Alert Microsoft Software Engineer Stopped A Major Cyber Attack

Kevin Roose By Kevin Roose Reporting from San Francisco April 3, 2024 Leer en español The internet, as anyone who works deep in its trenches will tell you, is not a smooth, well-oiled machine. It’s a messy patchwork that has been assembled over decades, and is held together with the digital equivalent of Scotch tape and bubble gum. Much of it relies on open-source software that is thanklessly maintained by a small army of volunteer programmers who fix the bugs, patch the holes and ensure the whole rickety contraption, which is responsible for trillions of dollars in global G.D.P., keeps chugging along. Last week, one of those programmers may have saved the internet from huge trouble. His name is Andres Freund. He’s a 38-year-old software engineer who lives in San Francisco and works at Microsoft. His job involves developing a piece of open-source database software known as PostgreSQL, whose details would probably bore you to tears if I could explain them correctly, which I can’t. Recently, while doing some routine maintenance, Mr. Freund inadvertently found a backdoor hidden in a piece of software that is part of the Linux operating system. The backdoor was a possible prelude to a major cyberattack that experts say could have caused enormous damage, if it had succeeded. ADVERTISEMENT SKIP ADVERTISEMENT Now, in a twist fit for Hollywood, tech leaders and cybersecurity researchers are hailing Mr. Freund as a hero. Satya Nadella, the chief executive of Microsoft, praised his “curiosity and craftsmanship.” An admirer called him “the silverback gorilla of nerds.” Engineers have been circulating an old, famous-among-programmers web comic about how all modern digital infrastructure rests on a project maintained by some random guy in Nebraska. (In their telling, Mr. Freund is the random guy from Nebraska.) In an interview this week, Mr. Freund — who is actually a soft-spoken, German-born coder who declined to have his photo taken for this story — said that becoming an internet folk hero had been disorienting. Five U.S. Cases Targeting Big Tech Card 1 of 5 For years, Apple, Google, Meta and other tech giants operated unfettered. Now, regulators around the world are taking steps to compel them to make major shifts to their products and businesses. In the United States, several cases could result in orders for the companies to change their practices. Here’s what to know. Apple. The Justice Department joined 16 states and the District of Columbia in filing a lawsuit against Apple, arguing that it had violated antitrust laws with practices that were intended to keep customers reliant on their iPhones by preventing other companies from offering applications that compete with Apple products. Amazon. The Federal Trade Commission and 17 states have sued the e-commerce giant, accusing it of protecting a monopoly over swaths of online retail by squeezing merchants and favoring its own services. The outcome could alter the way Americans shop online. Meta. The Federal Trade Commission and more than 40 states have accused Facebook of buying up its rivals, particularly Instagram in 2012 and WhatsApp two years later, to illegally squash competition that could have one day challenged the company’s dominance. They have called for the deals to be unwound. Google. The Justice Department has targeted the tech giant with two separate lawsuits. In 2020, the agency accused Google of illegally protecting its monopoly over search and search advertising. In 2023, the Justice Department and a group of eight states accused it of illegally abusing a monopoly over the technology that powers online advertising. “I find it very odd,” he said. “I’m a fairly private person who just sits in front of the computer and hacks on code.” The saga began earlier this year, when Mr. Freund was flying back from a visit to his parents in Germany. While reviewing a log of automated tests, he noticed a few error messages he didn’t recognize. He was jet-lagged, and the messages didn’t seem urgent, so he filed them away in his memory. But a few weeks later, while running some more tests at home, he noticed that an application called SSH, which is used to log into computers remotely, was using more processing power than normal. He traced the issue to a set of data compression tools called xz Utils, and wondered if it was related to the earlier errors he’d seen. Editors’ Picks 5 Minutes That Will Make You Love Shirley Horn For $367 a Month, She Has 345 Square Feet, a Hot Tub and a River The Case Against ‘Good’ Coffee SKIP ADVERTISEMENT ADVERTISEMENT SKIP ADVERTISEMENT (Don’t worry if these names are Greek to you. All you really need to know is that these are all small pieces of the Linux operating system, which is probably the most important piece of open-source software in the world. The vast majority of the world’s servers — including those used by banks, hospitals, governments and Fortune 500 companies — run on Linux, which makes its security a matter of global importance.) Like other popular open-source software, Linux gets updated all the time, and most bugs are the result of innocent mistakes. But when Mr. Freund looked closely at the source code for xz Utils, he saw clues that it had been intentionally tampered with. Kevin Roose and Casey Newton are the hosts of Hard Fork, a podcast that makes sense of the rapidly changing world of technology. Subscribe and listen. In particular, he found that someone had planted malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine. In the cybersecurity world, a database engineer inadvertently finding a backdoor in a core Linux feature is a little like a bakery worker who smells a freshly baked loaf of bread, senses something is off and correctly deduces that someone has tampered with the entire global yeast supply. It’s the kind of intuition that requires years of experience and obsessive attention to detail, plus a healthy dose of luck. At first, Mr. Freund doubted his own findings. Had he really discovered a backdoor in one of the world’s most heavily scrutinized open-source programs? ADVERTISEMENT SKIP ADVERTISEMENT “It felt surreal,” he said. “There were moments where I was like, I must have just had a bad night of sleep and had some fever dreams.” But his digging kept turning up new evidence, and last week, Mr. Freund sent his findings to a group of open-source software developers. The news set the tech world on fire. Within hours, a fix was developed and some researchers were crediting him with preventing a potentially historic cyberattack. “This could have been the most widespread and effective backdoor ever planted in any software product,” said Alex Stamos, the chief trust officer at SentinelOne, a cybersecurity research firm. Inside the World of Tech Stopping a Huge Cyberattack: A Microsoft engineer noticed something was off on a piece of software he worked on. He soon discovered someone was probably trying to gain access to computers all over the world. ‘Twitter Menace’ or True Believer?: The deep-pocketed tech investor Garry Tan says he wants to save San Francisco. But his pugnacious online habits are making him enemies. E.U. Investigations: Alphabet, Apple and Meta are under investigation for a range of potential violations of the E.U.’s new competition law. The inquiries signal the bloc’s intention to tightly enforce the sweeping rules that recently took effect. A New Antitrust Case: The Justice Department joined 16 states and the District of Columbia to file a significant challenge to the reach and influence of Apple, arguing in a new lawsuit that Apple had violated antitrust laws. If it had gone undetected, Mr. Stamos said, the backdoor would have “given its creators a master key to any of the hundreds of millions of computers around the world that run SSH.” That key could have allowed them to steal private information, plant crippling malware, or cause major disruptions to infrastructure — all without being caught. (The New York Times has sued Microsoft and its partner OpenAI on claims of copyright infringement involving artificial intelligence systems that generate text.) ADVERTISEMENT SKIP ADVERTISEMENT Nobody knows who planted the backdoor. But the plot appears to have been so elaborate that some researchers believe only a nation with formidable hacking chops, such as Russia or China, could have attempted it. According to some researchers who have gone back and looked at the evidence, the attacker appears to have used a pseudonym, “Jia Tan,” to suggest changes to xz Utils as far back as 2022. (Many open-source software projects are governed via hierarchy; developers suggest changes to a program’s code, then more experienced developers known as “maintainers” have to review and approve the changes.) The attacker, using the Jia Tan name, appears to have spent several years slowly gaining the trust of other xz Utils developers and getting more control over the project, eventually becoming a maintainer, and finally inserting the code with the hidden backdoor earlier this year. (The new, compromised version of the code had been released, but was not yet in widespread use.) Mr. Freund declined to guess who might have been behind the attack. But he said that whoever it was had been sophisticated enough to try to cover their tracks, including by adding code that made the backdoor harder to spot. “It was very mysterious,” he said. “They clearly spent a lot of effort trying to hide what they were doing.” ADVERTISEMENT SKIP ADVERTISEMENT Since his findings became public, Mr. Freund said, he had been helping the teams who are trying to reverse-engineer the attack and identify the culprit. But he’s been too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is coming out later this year, and he’s trying to get some last-minute changes in before the deadline. “I don’t really have time to go and have a celebratory drink,” he said. Kevin Roose is a Times technology columnist and a host of the podcast "Hard Fork." More about Kevin Roose A version of this article appears in print on April 4, 2024, Section A, Page 1 of the New York edition with the headline: Spotting a Bug That May Have Been Meant to Cripple the Internet. Order Reprints | Today’s Paper | Subscribe