Anthropic’s New AI Model Can Identify More Software Bugs Than Ever. Microsoft Is Struggling To Fix Them Fast Enough.
from the keeping-up-with-the-ai dept
This story was originally published by ProPublica. Republished under a CC BY-NC-ND 3.0 license.
On an afternoon in mid-May, dozens of Microsoft engineers and their managers gathered online and in a conference room at the company’s Redmond, Washington, headquarters to discuss Project Glasswing.
The tech giant was racing to fix weaknesses in its code that a new AI model known as Mythos was uncovering at an unprecedented clip. The AI behemoth Anthropic, which developed Mythos, had given access to select organizations that make software used by regular people, companies and governments across the world. The goal was to find and fix the vulnerabilities before hackers and adversarial governments like China began using similar tools to find and exploit them for espionage and sabotage.
As the group settled in, one engineer asked the question that loomed over the meeting: Did Mythos “live up to the hype that Anthropic claimed it would have had?”
“Yes,” a manager responded, according to a recording of the meeting viewed by ProPublica.
The version being used by Microsoft, Claude Mythos Preview, was surfacing bugs faster than the tech giant could patch them, and engineers, the manager said, were now in “a mad dash” to close the gap.
One slide in that day’s presentation showed that in April alone, Mythos had uncovered 90 “critical” bugs and 141 “important” ones in SharePoint, Microsoft’s widely used collaboration software. In the first half of May it found even more.
“Please, please, please if your org has any April bugs, drive those down,” engineering manager Hans Andersen implored the group. They had roughly two weeks “to find as many things and do as much good as we can with this access.”
May 31, he explained, “is considered the day when the rest of the world will have caught up.”
The engineers on the call poked at that assertion, with one of them summing up the predicament: “So basically you’re saying if it’s released on June 1, then on June 2 the adversaries will have our bugs?”
Yep, one person responded. Yep, another echoed.
Ever since Anthropic kick-started a national conversation about the bug-hunting power of AI in April, when Project Glasswing was made public, national security experts predicted that the U.S. would have a window of opportunity to fix flaws before adversaries would have similar models capable of discovering the same weaknesses. In late June, the international alliance of intelligence agencies known as the Five Eyes — whose members are the U.S., Australia, Canada, New Zealand and the U.K. — warned in an unusual joint statement that in a matter of months, that window would be closing. But the recording of the Microsoft meeting, along with internal documents reviewed by ProPublica, suggest the day of cyber reckoning may already be here.
Given the deluge of flaws Mythos has identified, Microsoft so far has focused on patching those it considers most dangerous, which are classified critical or important, according to the presentation as well as the company’s own public patch updates. The internal records indicate that Microsoft plans to eventually address “moderate”-severity flaws uncovered by Mythos. The documents made no mention of “low”-severity bugs.
The company’s approach reflects the triage system that is typical in the industry. Just as the sickest patients are the first to be treated in the emergency room, vulnerability triage prioritizes issues that are likely to cause the most damage if exploited by hackers.
But that strategy carries its own risk in this AI-powered bug-finding era, in which new tools are unearthing a record-breaking volume of weaknesses in the products we use every day. Mythos, for example, is able to chain together a string of bugs that build on one another, meaning that the low- and moderate-severity vulnerabilities that remain unpatched could create an opening to carry out devastating attacks.
“The problem now is that you can chain four low-level flaws, and that can equal a high severity,” said Vinh Nguyen, a senior technical adviser to Anthropic and a senior fellow for AI at the Council on Foreign Relations who formerly served as chief AI officer and chief data scientist at the National Security Agency. “If you’re Microsoft, the current triage strategy may be underpricing risks.”
In emailed responses to ProPublica’s questions, Microsoft stood by its approach, saying its triaging decisions are based on a number of factors, including exploitability and the impact on customers. The company presentation did not mention chaining, but a spokesperson told ProPublica that the technique “has long been considered as part of vulnerability assessment and risk analysis.”
Asked about the internal presentation and the then-looming May 31 deadline, the spokesperson downplayed its significance, saying that “accelerated targeting and exploitation of new vulnerabilities is not a new phenomenon.” That said, he added, the comments made during the meeting reflect how the company “feels a sense of urgency to help our customers at this time.”
“What was heard on that call and is true today is that security is Microsoft’s most important priority and teams across the company are prioritizing using AI to discover and remediate vulnerabilities as quickly as possible.”
Microsoft declined to answer questions about how many bugs engineers had patched since the presentation.
Anthropic declined to comment.
The internal Microsoft presentation and accompanying slides predicted that the group of staffers working on SharePoint, which is used by governments and businesses worldwide to manage data and documents, “will be busy for months,” first working through the highest-priority critical bugs then tackling the important ones in August. Microsoft says vulnerabilities it categorizes as critical include so-called worms that can crash systems and spread malware as they race across computer networks. Important ones could result in “compromise of the confidentiality, integrity, or availability of user data” as well as the “availability of processing resources.” After those categories were cleared, the group would begin work on roughly 300 “moderate” bugs, according to the presentation.
While the internal documents reviewed by ProPublica do not include updates on the entire breadth of Microsoft’s offerings, they do give a sense of the scale of the problem. One document noted that, since the company started using Mythos earlier this year, it had collectively found hundreds of bugs that Microsoft categorized as either critical or important in popular products such as Microsoft 365, the Teams conferencing platform and the Copilot AI tool. As of mid-May, most of them had yet to be patched.
“They’re not profound and exotic, but they’re real,” Andersen, the engineering manager, said during the meeting. “And a lot of them are exploitable.”
It’s unclear whether hackers have exploited any specific bug identified by Mythos, but some have tapped AI to automate attacks and appear to be using Mythos-like tech to find and exploit weaknesses.
There have been outward signs of Microsoft’s internal struggle to deal with the growing list of bugs to be patched. Each month, the company publicly releases fixes for its software vulnerabilities in what’s known as “Patch Tuesday.” In June, it released patches for more than 200 bugs, which industry experts then said was an all-time high. But on July 14, the company blew through that record and released patches for more than 600 bugs. Only seven were categorized as low- or moderate-severity, one of which hackers were actively exploiting, according to Dustin Childs, leader of the Zero Day Initiative bug bounty program, which is part of cybersecurity company TrendAI. The rest were important or critical.
“Well folks. Here we are. The bug apocalypse has fully descended upon us,” Childs wrote in a blog post on July 14.
Microsoft told ProPublica that the overall volume of bugs “will not be plateauing for a bit,” but a spokesperson said the company has “invested heavily in both people as well as AI-powered triage solutions that scale quickly to handle the growing number of vulnerabilities.”
Given the new realities of the AI age, including the chaining capabilities, companies like Microsoft might need to rethink their entire approach to triage, said Nguyen, the NSA’s former AI chief. Rather than shunting what are now considered low-risk flaws aside, companies should be dedicating staff to developing and testing patches for the entire spectrum of vulnerabilities, he said. In other words, the cyber ER needs more doctors and nurses treating illnesses that are life-threatening as well as the minor wounds that could later turn deadly.
“There’s no alternative,” Nguyen said. “The patients are coming in fast and furious.”
Microsoft told ProPublica it’s “always going to be reevaluating and considering whether things that were previously lows or moderates be upgraded or thought about differently. With these AI systems, it makes us rethink some of these things. Across the industry, we’re all looking to see how drastic of a change it will be.”
Microsoft’s users may be particularly vulnerable. The popularity of its offerings, used the world over, makes it a frequent and lucrative target for hackers. In addition, many of its products contain “legacy” code. Developed decades ago using now-outdated technology, this code contains unaddressed flaws and contributes to what is known in the industry as “technical debt.”
But the challenge of fixing the flood of newly found bugs also extends to the rest of the software industry, and to open-source software code that is typically free to use and largely maintained by volunteers. Open-source software underpins internet infrastructure and is incorporated into much of the world’s modern technology, including products offered by major tech companies such as Microsoft.
“Nobody has really figured out how to deal with this, and everybody is casting around for what they need to do,” said J. Michael Daniel, a former cybersecurity adviser to President Barack Obama and the president of the Cyber Threat Alliance, a nonprofit organization focused on cybersecurity. “Our tech debt is coming due.”
Ben Edwards, a data scientist who specializes in managing software vulnerabilities, said the software industry was handling an “intense volume even before AI.”
“It was like drinking from a garden hose on the jet setting before, and now it’s like drinking from a fire hose,” Edwards said. “They might have had the teams that could handle that garden hose. Whether they can handle the fire hose is something else.”
Although the volume of vulnerabilities has grown over the years, Microsoft’s internal group responsible for fielding them, the Microsoft Security Response Center, has been perennially understaffed. Even before the crush of AI-identified bugs, the center fielded hundreds or even thousands of reports a month, pushing the group to its limits, ProPublica has reported.
The size of the center reflects Microsoft’s corporate philosophy: Plugging security holes is a cost center, while making new products is a profit center, former employees said. The company is loath to tie up its best engineers with making security patches — a cost center — instead of developing new products and features that will generate profits, ProPublica has reported.
Microsoft told ProPublica that it does not discuss internal staffing decisions but has made investments in recent years to “focus our teams on keeping our customers secure.” The company “continuously evaluates the staffing, processes, and technologies required to support security response and vulnerability management,” a spokesperson said.
According to the slides that accompanied the May internal presentation, Anthropic provided Mythos access to roughly 50 full-time Microsoft employees, with a goal to “harden critical services before publicly available models catch up.” A slide titled “What’s Next” predicted that the Microsoft Security Response Center would see continued case volume “as public tools catch up” to Mythos.
During the May meeting, one staffer appeared to take comfort in the belief that adversaries “don’t have the source code” that such an AI tool would scan for weaknesses. His colleagues, however, quickly corrected him. Portions of Microsoft’s code have, in fact, fallen into hackers’ hands over the years.
“It might not be this week’s source code,” one person said. “But they’ve got source code. It’s out there.”
In a statement to ProPublica, Microsoft downplayed the comment, saying engineers “design our security processes on the expectation that determined adversaries may gain access to code.”
Filed Under: ai, bug hunting, bug patching, cybersecurity, mythos, security, vulnerabilities
Companies: anthropic, microsoft
Comments on “Anthropic’s New AI Model Can Identify More Software Bugs Than Ever. Microsoft Is Struggling To Fix Them Fast Enough.”
Either this is a misreading of the page, or refers to an old version of it. The current text is “A vulnerability whose exploitation could allow code execution without user interaction. These scenarios include self-propagating malware (e.g. network worms)”… that is, a worm is not a vulnerability, but something that can take advantage of certain vulnerabilities—those vulnerabilities being “critical”.
In this analogy, while it’s true that sick people need to be treated, we also need to be talking about prevention, which this article says very little about.
The old model of computer security was basically “we’ll hack out some code that mostly works, fix whatever problems we find later, and hope no bad people find major security bugs before we do”. It was never a very good model, which is why nation-state attackers always seem able to find a vulnerability: they have the resources to actually look for them. People learned the wrong lesson from the 1982 “E.T.” video game.
Formal verification is probably the best way forward. The developers of the seL4 operating system kernel managed to create a security proof by throwing wave after wave of grad students at the problem. I believe the process was helped by ARM releasing machine-readable specifications of their processor behavior; nevertheless, the proofs still have various assumptions and limitations, such as assuming the processors behave as documented (so let’s hope ARM and Intel also make good use of formal verification).
An optimistic prediction is that automated computer systems might be able to bring formal verification into reach for everyone. But that still leaves the question of what to do about pre-existing software that’s not well-designed for formal verification, or just has way too many bugs for humans to fix.
Re:
This is true, although it was likely out of the scope of the meeting ProPublica was able to view.
Working in SW verification, in my opinion, they would need to start incorporating these scans at minimum every time they want to push out a new patch/release, and likely at regular intervals as the AI models themselves are updated – and then do similarly with new software as it’s being developed. How often it can be done likely depends on how long these scans take, but absolutely incorporating this into their (anyone’s, not just MS, really) release processes, alongside unit tests, formal validation/verification, etc., could certainly be beneficial.
Re: Re:
A pre-release scan will only find mistakes that have already happened. Better than finding them post-release, of course, but scans and such seem like an attempt at extending a fragile development model.
Testing or verification should not be treated as a separate activity from design and coding, with defect reports and scan results as the main communication channel. For one thing, the root cause of software defects is sloppy thinking in the first place; if the programmers are incapable of describing the conditions for correctness, how’s any human or computer gonna figure it out after the fact? Stuff will necessarily be missed.
(See also “No Silver Bullet” by Frek Brooks. New tools often help, but not nearly as much as the hypesters expect.)
Re: Re: Re:
Correction: Fred Brooks (famous for “The Mythical Man-Month”); here’s the paper: No Silver Bullet—Essence and accident in software engineering (1986)
Thanks, tech assholes.
Soon, Microsoft might have to stop shoving telemetry and Copilot into every nook and cranny to unfuck the mess they made shoving telemetry and Copilot into every nook and cranny.
Re:
Or they’ll add more telemetry, to see which defects people are exploiting. In some cases, they’ve already found and fixed stuff based on crash dumps. (They’ve also found people with unreliable RAM, but haven’t done anything about that.)
I don’t really see how cramming telemetry and Copilot into things has much to do with their current situation, except that the people assigned to that work could’ve been doing useful work instead (like spending more time to create better designs, or reviewing existing code for problems).
Re: Re:
“I don’t see how this would have helped, except where it would have helped.”
Re: Re: Re:
Except not “would have”, because if Microsoft’s management doesn’t see bug-fixing as “useful”, why would they assign a newfound surplus of programmers to that? Laying them off, instead, would “save” the company many millions of dollars per year, and who ever got fired for suggesting that?
I’d be impressed if it fixed bugs.
Re:
For sure, it fixed bugs. Microsoft spends hard cash on competent engineers.
But they’ve also introduced new and more complex bugs, due to Microsoft internal structure, engineers only work on a very specific part of a specific product, not knowing the complete scope of the problem.
Sure, if Microsoft hasn’t been forcing its whole ecosystem even just to edit a docx file, theses vulnerability chaining would be much harmless.
But if not like Microsoft has a good reputation on security.
Re:
It absolutely can do this, how do you think they are fixing the bugs?
The difference is that the prompts were to find bugs, not to fix them.
““What was heard on that call and is true today is that security is Microsoft’s most important priority…”
Nah! What MS is about is maximising the extraction of data and profit from its victims. Fixing bugs costs money and that means that MS will always under invest in bug-hunting.
Time to get off the MS train.
AI to exploit and to patch
The security cat and mouse game continues as tech companies are also using to AI to patch these vulnerabilities.
The chaining attack is a known but lesser used attack method because it was so (human) resource intensive without certainty it would succeed.
It’s also an unfortunate balance tech companies must achieve because if they simply used all their resources to patch vulnerabilities, they’d have few resources to develop new features (putting aside their profitability).
Most companies that are part of project glasswing are in the same boat.
Re:
Of course, both finding and patching require some knowledge of the correct behavior. Had the programmers been diligent about defining such things, the written code would likely have reflected it.
To the computer, it’s all good all the time. We could ask an automated system to find instances of stack overflows, for example. But, to the processor, it’s working as designed; after all, if a stack adjustment might’ve caused undesirable behavior, the programmer would not have ask the computer to do it without checking, and maybe would’ve reprogrammed their compiler to add checks.
So, the “fixed” program will have observably different behavior than the original, and who’s to say which is better? It’d be easy to say that any difference visible to Microsoft Excel would be unacceptable, but if a spreadsheet authored by NSO Group got into the data set used for testing, that might be something we want to “break”. Then again, Linux was once patched to restore compability with an old virus, on the basis that the virus was using the interface as intended and the breakage was not intentional.
What about us individual developers?
Many of us have open-source projects which we would love to have analyzed. But we have no indication such is going to happen. And we cannot afford to pay anyone to do the AI analysis. So what happens now?
Re:
Be careful what you wish for. If your project gets used by something or someone “important”, like Windows or Android, it’ll get analyzed. Certainly by malware authors. Perhaps by the vendors using the code, who might expect you to act like a major organization but for free; they’re probably not gonna do anything beyond filing the report and pestering you about its status.
Developers of projects including curl have complained about a flood of slop: auto-generated “security” reports whose impact is usually exaggerated, when the report is not outright bullshit or a hallucination.
adversarial government
You forgot to also include the USA as one of the examples for adversarial governments.
Of course you can just put every government down as an example because they all are adversarial to their own people or those of another country to some degree.