A week ago, on March 29th, the U.S. Cybersecurity and Infrastructure Security Agency (CISA) warned that two versions of xz Utils, were found to have been compromised. The xz Utils code had been tampered with to include a malicious “backdoor” that would ultimately give attackers the same level of control over affected systems as authorized administrators. Xz Libs is a set of open source libraries widely-used as dependencies for Unix operating systems, including Linux—the operating system supporting most of the world’s server infrastructure.
On the same day of the CISA warning, open source software provider Red Hat sent out an urgent security alert advising users that two of their Linux ecosystem products were affected by the xz Utils malware. “Immediately stop using Fedora 40 or Fedora Rawhide until you can downgrade your xz version,” the alert noted.
Astonishingly, the attack appears to have been as long as three years in the making. In 2021, a GitHub account was created by an unknown person or organization. The entity behind the account gradually became a co-maintainer of the xz Libs as the original maintainer, citing overwork and burnout, ceded primary oversight of updates.
In early 2024 malicious code linked to the new maintainer was inserted into the xz Libs project. “This might be the best executed supply chain attack we’ve seen … it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” wrote another open source maintainer. Had the backdoor remained undetected, said computer scientist Alex Stamos, “it would have given its creators a master key to any of hundreds of millions of computers around the world…”
Maintainer burnout is a dangerous problem impacting many open source projects, as it apparently did with xz Libs. Of the 900+applications that included risk assessments examined in the OSSRA report, 49% contained open source that had no new development activity in the last two years.
If a project is no longer being maintained that means there have been no feature upgrades, no code improvements… and no security issues fixed. Worse, as the xz Libs attack shows, attackers may weasel their way into trusted roles in popular projects with declining maintenance and eventually have the opportunity to insert tainted versions of the project into the supply chain stream.
“Zombie” code—in a twilight, unmaintained state, but still in use, is not an uncommon problem for open source projects which largely depend on unpaid, volunteer maintainers to keep the projects safe and updated. According to some reports, nearly 20% of Java and JavaScript open source projects maintained in 2022 were no longer being maintained in 2023, opening those projects to abuse and exploits.
When an open source dependency becomes popular, the price is increased pressure on its (usually unpaid) maintainers—the people who handle bug reports, feature requests, code reviews, and code commits for the free software. It’s not unusual for the “maintainers” to actually be a solo developer—a “random person from Nebraska,” as the popular xkcd internet comic has it.
That random person is often the only bulwark supporting their piece of the open source infrastructure that modern software depends on. As an open source project grows in popularity—with no corresponding growth in people maintaining the project—the consequence is often developer burnout, and many open source projects are abandoned.
One possibility contributing to this trend is that the growing number of security vulnerabilities is making it so difficult for overworked maintainers to keep code updated that they abandon projects rather than continuing efforts which will never have conclusion. Indeed, the OSSRA report states that 74% of the applications scanned for security risk in 2023 contained high-risk vulnerabilities, a significant increase from previous years. According to cybersecurity research by Qualys, over 26,000 vulnerabilities (including those impacting open source and other third-party code) were disclosed in 2023, 1,500 more than the previous year.
Unfortunately, erratic maintenance and too many open source consumers’ willingness to download code without checking for potential security issues have made open source repositories a favored vector for attackers.
As the popularity and use of open source has grown, open source risk has evolved, from licensing compatibility issues, through opportunistic exploits of vulnerabilities, to malicious attacks. Planned, deliberate attacks on software supply chains are now common, with malicious actors moving beyond taking advantage of code weaknesses to actual tampering with code and development pipelines to spread tainted malware. In its 2024 report, “The State of Software Supply Chain Security,” ReversingLabs cites a more than 1,300% increase within the past three years in threats originating from open source package repositories, including incidents where attackers pushed malware directly through software packages. Two popular open source software platforms have had a 400% increase alone in malicious PyPI packages (third-party software for the Python programming language) just in the last year.
Organizations that use open source in their software— which is literally all organizations—need to proactively identify and manage these threats as part of securing their software supply chain.
What can you do to protect yourself?
-This blog was fact checked by Mike McGuire.