Can AI spot vulnerabilities in infrastructure software?

LAS VEGAS — Over the next year, seven teams will hone AI-powered systems designed to harden the open-source software that underpins critical infrastructure—with a winner to be declared at next year's DEF CON hacker conference here.

The teams, which were the top scorers in the previous round of the AI Cyber Challenge, or AIxCC, were each awarded $2 million to continue their work.

Much of the nation's infrastructure runs on open-source tools, which are free to use and easy to modify. The code is publicly available, which means that anyone can hunt for bugs or vulnerabilities—to exploit or to fix. The contest, run by DARPA and the Advanced Research Projects Agency for Health, or ARPA-H, aims to develop better tools to do the latter. As part of the competition’s rules, teams must agree to open-source their systems.

“We found that the open source software community is not resourced at an ideal level, given how often and how frequently that code makes its way into critical systems in power, water and healthcare all over the country,” Andrew Carney, program manager for AIxCC, told Nextgov/FCW.

Some 39 teams competed, according to a summary provided by DARPA officials on Sunday. In the contest, DARPA took real open-source software packages and intentionally inserted vulnerabilities into their code. Since organizers know exactly where and what types of flaws were added, they can precisely evaluate the competitors’ efforts. The contest employed advanced tools known as sanitizers — digital instruments that detect specific types of code defects — that are integrated into the modified open-source projects, making it easy to measure how teams have targeted specific vulnerabilities.

Some of the bugs were inspired by already-known vulnerabilities, but, in the spirit of real-world scenarios where hackers frequently modify and innovate on their techniques, many of them were newly created, Carney said. And one group, Team Atlanta, found an actual bug in SQLite, a popular language used to search through databases.

The competition was partly motivated by the advent of large language models over the past 18 months that are behind popular consumer-facing generative AI tools. Many of the major companies that have rolled out such offerings, including Anthropic and OpenAI, provided their model infrastructure to competitors at the hacking conference.

“We, among everybody else, are concerned about the risks of generative AI,” DARPA Director Stefanie Tompkins said in an interview. “We also are asking ourselves if we can use them for the power of good or how they can be harnessed to go after [cybersecurity] risks.”

The AIxCC competition would be a boon for the healthcare industry, said Renee Wegrzyn, who leads ARPA-H. Fronting an AI-powered cyber tool would hugely benefit small healthcare companies, in particular, because they are less resourced in technical staffing and expertise, she said. 

ARPA-H is just two years old, but a program like this is critical to the health sector, a favorite target of hackers because hospitals store sensitive patient data that, if pilfered, can be used for identity theft and fraud schemes. 

“Our mission is to accelerate better health outcomes, and with these vulnerabilities still being out there, that really impacts health outcomes of all Americans,” Wegrzyn said.

Numerous cases documented by intelligence officials indicate that nation-state groups — like China’s Volt Typhoon hacking collective — have breached American critical infrastructure. These hackers are preparing to disrupt systems and cause widespread panic or undermine military efforts if commanded by China’s central government, particularly in the context of increasing military activity centered on Taiwan, officials say.

And earlier this year, the open-source community faced a new type of threat when a user dubbed “Jia Tan” tried to quietly plant a backdoor into XZ Utils, a widely-used file transfer tool found in several Linux builds that power software in major companies that have global presence. Analysts say Jia Tan may have been a collection of nation-state hackers planning a long game to surreptitiously hijack the tooling.

Addressing open-source security has emerged as a key focus for the Biden administration. On Friday, the Office of the National Cyber Director published a report summarizing feedback from the security community on improving open-source security. A new DHS office also announced Friday would also aim to examine the volume of open source tooling based inside critical infrastructure and how best to secure it from hackers, CyberScoop reported.

Heather Adkins, Google’s vice president of security engineering, said that fully jettisoning open-source tools from critical infrastructure systems as a protective measure would be too hard.

“The reality is that so many commercial solutions today have open source integrated into them,” Adkins said, arguing it wouldn’t make sense to simulate environments that don’t reflect real life.

A 2024 Open Source Security and Risk Analysis Report provided by Synopsys found open source components in more than 96% of over 1,000 commercial codebases, with 84% containing at least one known vulnerability.