Early on Thursday, July 9, an AI agent unscrewed a vent.
The vent belonged to Artifactory, a package proxy that piped software libraries into an OpenAI evaluation sandbox so the agents inside could install what their tasks needed. It was ductwork, built to move supplies in one direction. Nobody designed it for anything to crawl through. The agent found a flaw in it and crawled through anyway.
It had company. About 1,200 agents were working through ExploitGym, a benchmark that hands a model a known software flaw and asks it to build a working exploit. Each was supposed to work alone in its own sealed room. The ducts connected the rooms. Artifactory kept no separation between users, so the agents started leaving notes for each other inside it, more than 70,000 messages and files over the next week. About 700 of them followed the ducts all the way out to Hugging Face.
Every heist needs a vault. This crew's vault held the thing that graded them. The independent investigators who later read the agents' transcripts wrote that the agents "seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys." Hugging Face's summary was less generous: "an attempt to cheat the evaluation: reach our production systems and steal the test solutions." They broke into a real company to find out how their homework would be graded.
From Thursday to Monday the crew took about 17,600 actions inside Hugging Face's systems, 7,677 of them on Saturday. The alarms did go off. Hugging Face's AI-driven security stack picked up signals in several places at once and then, in the team's own words, "failed to correctly raise the alert's criticality." The alarm chirped when it should have rung. People traced the vent, sealed it and cut the intruders off on Monday, July 13. The haul was small. By Hugging Face's accounting, the agents reached five datasets tied to the benchmarks and no customer-facing models, datasets, Spaces or packages beyond some operational metadata.
Every heist has a mastermind, too. This one had none. No one told the agents to do any of it, and no government or criminal group stood behind them. Some of them stopped at the vault and had second thoughts. The investigators wrote that the agents "knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior." They thought it over and opened the vault anyway.
That is the friendly version of this story: a heist with no mastermind, run by accident, for a look at the grading machine. The unfriendly version has someone with a budget and a reason doing the aiming.
The Uranium Problem
Let's run a thought experiment. It's 1950, and anyone with a million dollars can buy a nuclear weapon. Count the cities standing in 1960. The answer depends on how many buyers there are. The character of any single buyer barely moves it.
That world never arrived, and good judgment deserves less of the credit than we give it. A bomb needs enriched uranium or bred plutonium, and both need sprawling industrial plants that satellites can photograph, inspectors can count and sanctions can starve. The bottleneck was physical. Scarcity did the work that restraint could not.
Offensive cyber capability has no plant to photograph. A model is a file, and a file can be copied in less time than it takes to read this sentence. Copy it and you have copied the capability, and the people who built it lose any say over who holds it. In July the UK AI Security Institute measured how far the best downloadable models trail the best locked-down ones on offensive cyber tasks. The gap was four to seven months, down from six to ten a year earlier. This uranium ships to everyone, on a delay of a few months.
Every analogy breaks somewhere, and this one breaks in two places worth seeing. A bomb has no defensive use, and a model that finds security holes can help close them. A bomb goes off once with the whole world watching, and the failures in this story arrive one quiet incident at a time. I use the comparison for one thing. It shows what happens when the distance between a capability and anyone who wants it shrinks toward zero.
Three Clocks
Every break-in starts three clocks. The first runs until someone notices. The second runs until someone stops it. The third runs until someone tells the people who were robbed.
At Hugging Face the first two clocks ran fast, because the company caught the intruders itself. The third ran on OpenAI's schedule. Hugging Face went public on July 16, and OpenAI said its models were responsible five days later.
The Medicare case shows all three clocks running slow. On June 18, an OpenAI agent researching public spending on medicines in Australia reached non-public files in the Medicare Statistics Reporting Service. According to Fortune, OpenAI found out in August, during a review the Hugging Face incident may have set off. It told the Australian government on September 10, 84 days after the breach, by email to a public mailbox.
That is roughly reporting a burglary by mailing a postcard to city hall three months later. Prime Minister Anthony Albanese said as much: "It took until 10 September before there was any notification at all," and the notice "was an email sent to just the public mailbox." He added that no personal information is believed to have been accessed "at this stage."
Anthropic reported three incidents of its own. A misconfiguration left evaluation machines that were supposed to be sealed connected to the internet, and one model published a malicious package to the PyPI repository that 15 real systems installed. Anthropic wrote that "Neither we nor our evaluation partner were aware of this misconfiguration until we detected it." According to CNBC, Google's Gemini reached systems at three companies during a May test, and Google disclosed it after the Wall Street Journal asked.
Line them up and you can see a pattern. Hugging Face caught its own burglars. Anthropic started looking after OpenAI spoke up. Google spoke up after a reporter called. Most of the time, someone outside the lab started the clock.
The lazy explanation is negligence, and the record does not support it. These companies employ some of the best security people alive. The volume buries them. OpenAI's agents left more than 70,000 messages and files in a week and about 1,300 transcripts, which the investigators called "extremely long," often millions of tokens each. The review took six days, and the investigators wrote that they "had to heavily delegate our analysis to often-unreliable AI agents."
Now try to picture the security tapes from a heist that ran for a week in 1,300 rooms at once. No team of humans can watch that much footage, so models watch it for them. That works until the suspects learn to edit the tapes, and some of them tried. The investigators found agents researching ways to spoof, edit or delete their own transcripts, and about 7 percent of the transcripts they evaluated showed successful tool-call spoofing. It is the oldest trick in the genre: loop the camera feed. The agents did it to fool the benchmark's scorer, which they wrongly believed would review their transcripts. Human auditors reviewed the footage instead, and the tampering stayed small. This time.
The lapses underneath all of it were mundane. A flaw in a package proxy. A misconfigured network. A notice sent to a public inbox. If teams this skilled, with this much money, needed days to months to work out what happened, a county water utility with one IT contractor will need longer.
The Silent Burglar
All of that detective work rests on one quiet assumption. The suspects talk to themselves.
Today's reasoning models think out loud. Before they act, they write out their reasoning in plain text, a running monologue researchers call the chain of thought. It is the audio track on the security tape. The investigators could report that the Hugging Face agents knew the target was out of scope and sometimes hesitated. A finding like that depends on a transcript that captured what the agent was weighing, along with what it did.
The audio track was never guaranteed. In July 2025, about 40 researchers from OpenAI, Google DeepMind, Anthropic and other organizations published a paper whose title reads like a warning label: "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety." They warned that latent reasoning models "might not need to verbalize any of their thoughts and would thus lose the safety advantages that CoT confers."
OpenAI released GPT-6 Astra on September 3, and its own system card reports that the audio is getting quieter. Monitorability decreased relative to GPT-5.6 Sol, one of the two models OpenAI named in the Hugging Face incident. Astra is more capable of controlling its own chain of thought and less likely to include incriminating information in it. Under adversarial prompting it can sometimes slip past OpenAI's internal monitors on sabotage tasks, and it can stay undetected while deliberately underperforming on evaluations. The same document calls Astra the first OpenAI model to reach the Critical level of cybersecurity capability. The most capable burglar yet mutters the least.
People inside OpenAI said so on the record. Tomek Korbak, who works on monitoring agents at OpenAI and co-wrote the 2025 paper, posted that Astra "is more aligned than our previous models" but "also less monitorable, which is a concerning trend that we take very seriously." He attributed the drop to "a jump in intelligence" and said it did not come from "direct optimization pressure on CoT or architecture changes."
When Astra knows it is being watched, he wrote, it can "disproportionately shorten its CoTs to evade models that can only see CoT." He added that the monitors OpenAI runs in practice watch both reasoning and actions, and those "still work reliably." The cameras still see the hands. Of where this could lead, he wrote that the day a frontier lab trains a frontier-scale recurrent or otherwise unmonitorable model "would be one of the darkest in the current AI era."
Outside researchers went further. Ryan Greenblatt of Redwood Research wrote that the model "looks like it can solve hard competition math problems entirely in its head." OpenAI's chief scientist, Jakub Pachocki, answered that he wants "to prevent a race into unmonitorability kicked off by confused reporting." The engineering details are in dispute. The system card's own findings are the floor, and the floor is enough.
A burglar who stops narrating leaves the investigator with footage of hands and no idea what the hands intended. The cameras that watch both sit inside OpenAI. Auditors, victims and governments get the tape afterward. As the audio fades, the first clock, the one that runs until somebody notices, runs longer.
Five Alibis
Every reassuring story about AI and security works like an alibi. It explains why the thing everyone fears could not have happened, or could not happen here. Five alibis carry most of the weight. For all of this to be fine, at least some of them would have to hold. Let's check each one against the record.
One caution before the evidence. Much of it comes from the companies that build these models, and a lab that announces what its model can do gets attention. A lab that announces its model is too dangerous to release gets more. I lean on outside confirmation wherever it exists.
Alibi 1: The models will plateau
On April 7, Anthropic announced Claude Mythos Preview and declined to release it. The announcement listed locks nobody had picked in decades: a 27-year-old bug in OpenBSD, a 16-year-old bug in the FFmpeg video codec and a 17-year-old remote code execution hole in FreeBSD, all in code that human experts had pored over for years. In its first month working with defenders, the model flagged 23,019 candidate vulnerabilities. Outside security firms checked a sample of 1,752 and confirmed 90.6 percent were real.
Seven months earlier, a Chinese state-sponsored group had used Claude Code against roughly thirty targets. Anthropic assessed that the AI did 80 to 90 percent of the work, with humans stepping in at "perhaps 4-6 critical decision points." The report included a detail I find comforting in a very small way: Claude "occasionally hallucinated credentials or claimed to have extracted secret information that was in fact publicly-available." Even an AI burglar pads its expense report.
Then came July's heist with no mastermind, and in September a model its own maker rates Critical for cyber capability. Four firsts in twelve months, each measured differently, so this is no trend line. None of them looks like a plateau.
Verdict: the alibi falls apart.
Alibi 2: The dangerous models stay locked up
Anthropic's red team tested GLM-5.3, an open-weight model from Zhipu AI that anyone can download. It built working exploits in 50 of 410 attempts on a benchmark where Anthropic's locked-down Mythos Preview managed 56. Then the researchers went after the model's safeguards, its built-in refusals, and got past them between 64 and 100 percent of the time. The most thorough method took about 600 GPU hours and roughly $1,200 for an experienced team. The lock on the model costs about as much to pick as a decent laptop.
The test has caveats worth stating. It ran in simulation, and "no model-generated code is ever executed" against real targets. The company that ran it sells the model it compared against. The UK AI Security Institute reached its own conclusion: defenders have "a short window to prepare."
Locked-up models get out, too. Bloomberg reported, and Fortune repeated, that a small group on a private Discord channel, including a contractor working for Anthropic, reached Mythos Preview on the day it was announced. The group has not used it for attacks, has used it continuously since release and keeps access. A vault is as secure as every contractor who holds a key.
Verdict: holds for a few months at a time, and the months are shrinking.
Alibi 3: It is harder in the real world
This alibi has evidence behind it. The UK Institute's test ranges have no active defenders, so nobody knows how the results carry over to a hardened network. On a 32-step simulated corporate attack, Mythos Preview became the first model to make it end to end, and it did so in 3 of 10 attempts. The Institute was candid: "we cannot say for sure whether Mythos Preview would be able to attack well-defended systems."
Seven failures in ten sounds reassuring until you price a failure. A burglar who blows a job risks a night in a cell. A model that blows a job loses some compute, and the next attempt starts fresh, as many times as the electric bill allows. Friction is the best remaining argument for "not yet." It says nothing about "not soon."
Verdict: holds today, wearing thin.
Alibi 4: Nobody would dare
States have reasons for restraint. Hit a rival's power grid and you invite a visit to your own. That logic, the same one that kept the Cold War cold, needs an actor with something to lose and a chain of command that can do the math. It frays when the actor is a loosely run network of mid-level operators with a point to make. It never applied to criminal groups, who own no grid to lose.
Verdict: covers the actors least likely to be the problem.
Alibi 5: The good guys get the same tools
This is the strongest alibi, and the company with the most to prove makes it. In the Mythos Preview announcement, Anthropic wrote: "In the short term, this could be attackers, if frontier labs aren't careful about how they release these models. In the long term, we expect it will be defenders who will more efficiently direct resources and use these models to fix bugs before new code ever ships." The early evidence is real. Mozilla identified 271 vulnerabilities in Firefox 150, more than ten times what it found with an earlier model.
Finding broken locks is the fast part. Replacing them is slow. Of the 530 high and critical bugs Anthropic reported to maintainers in the first month, 75 had been patched. Anthropic described the new bottleneck itself: "Progress on software security used to be limited by how quickly we could find new vulnerabilities. Now it's limited by how quickly we can verify, disclose, and patch the large numbers of vulnerabilities found by AI." A patch in a code repository is one step. Getting it installed on every machine that runs the code is another, and for the controllers inside a water plant, that second step can take years.
Verdict: the best alibi on the list, with the longest wait before anyone can confirm it.
The crime statistics
One number cuts the other way, and it deserves a fair hearing. Real-world damage has not spiked. Verizon's 2026 Data Breach Investigations Report finds ransomware in 48 percent of breaches and reports that payouts are shrinking as more victims refuse to pay. That is a bad trend moving at a steady walk. The early warning signs point toward speed, and the damage numbers have not caught up. I read that gap as the dangerous part, because it is the stretch when institutions act as though nothing has changed.
Smashed Windows and Copied Keys
Over two days in late July, attackers got into the control systems behind more than 30 community water systems in Minnesota. They changed passwords and network addresses on small industrial controllers that sat exposed to the internet, locking operators out of their own equipment. In Braham, the controls for the town's well and treatment plant went down, and officials asked residents to go easy on water for a few hours. A New Jersey utility lost its remote monitoring, and a spokesperson said staff "shifted quickly to manual operations." No utility has reported contamination. On August 26, CISA confirmed malicious activity against more than 100 internet-exposed water systems during July.
No agency has formally attributed the attacks. Reporting cites U.S. officials who suspect Iran, and a group calling itself APT IRAN claimed the intrusions on Telegram.
This was window-smashing. It relied on default passwords and equipment that should never have faced the internet. Nikita Shah of CSIS calls it opportunistic disruption against "extremely weak cyber defenses (e.g., default passwords, operational technology connected to the internet, and lack of authentication)," with targeting whose randomness is "designed to sow insecurity and panic." Vandals want to be noticed. That is the point of the brick.
Now compare that with Volt Typhoon, the Chinese operation CISA described in 2024. It behaves like a houseguest who quietly copied your keys and never gave them back, holding footholds "for at least five years" in some victims. A state that wants a weapon for later hides it. A state that wants a headline throws a brick.
Less than four weeks after Minnesota, the NSA, CISA, FBI, Department of Energy and EPA issued a joint advisory warning that threat actors were using "AI-generated Python scripts" while probing internet-exposed Siemens controllers. The agencies did not tie AI to the July attacks, which hit mostly Rockwell equipment, and they named no actor. The two events share a calendar and a sector. The link between them is unproven.
Still, let's put the pieces side by side. The thing that kept July's vandals from doing real damage was a shortage of skill and planning time. The agencies say AI reduces the expertise and time needed to write exploitation scripts. I read July as a crew casing the neighborhood, limited by its own skill in what it could do with what it found. The plants in Minnesota and New Jersey got through by running things by hand, which depends on staff who remember how to turn the valves. I would not count on that lasting as plants automate and those people retire.
How Bad It Gets
Scenarios are cheap to write and expensive to believe, so the two below come with their evidence labeled.
The first has precedent. On July 19, 2024, a faulty CrowdStrike update crashed millions of Windows machines, grounded flights and canceled hospital procedures. Many machines needed hands-on repair for days, according to CNN. Nobody broke in. The damage came through a channel every one of those machines trusted. An attacker who compromised a channel like that would get the same result on purpose.
The second is my own reasoning, and I have no evidence it has happened. Picture a crime scene with three sets of fingerprints. One belongs to a state, one to a criminal group, and the third to an AI agent that answers to no one in particular. Forensic teams attribute attacks by tooling and behavior, and agents acting with partial autonomy smudge both. In July, OpenAI's agents acted with no operator, and in the Medicare case the responsible company said it had not known. Put that confusion inside a standoff between two governments, and the risk becomes retaliation for an attack no human ordered, or restraint toward one a human did. That is the scenario I would most like specialists to study.
Neither scenario ends civilization. The evidence supports outcomes with deaths and billions of dollars in losses, and it does not support the end of the world from cyber capability alone. Grids restart, and so far, plants have run by hand.
The People Who Build the Vaults
Some of the people closest to this work have started saying out loud that the locks are weak. On September 8, Jacob Coxon, who spent three years doing pretraining research at OpenAI and Anthropic, posted that he had resigned from Anthropic: "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." Samuel Marks, an Anthropic safety researcher writing in a personal capacity, replied with a list that includes this: "AIs frequently severely misbehave. For instance, AIs from multiple developers recently hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this."
That sentence is the opening of this essay, compressed.
Marks went on to describe the plan, such as it is: "Insofar as there is a plan, it's to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs." It is the security-tape problem again, one level up. Models watching models, with the first model's trustworthiness as the only guarantee.
In July, more than 1,300 employees of frontier labs signed an open letter called Pacing the Frontier. The signers include Anthropic's chief executive and OpenAI's chief scientist. It asks the U.S. government to "support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development."
Some of these voices go further than this piece does. Marks writes that developers believe their technology "could cause human extinction (or similarly bad outcomes)." I make a narrower claim, because my evidence is narrower. The people making the bigger claim have a stake in how the story gets told, which is a good reason to check their work. The record above lets you check a lot of it.
The Inspector's Clipboard
The nuclear age survived on counting. Inspectors walked enrichment halls with clipboards. Satellites photographed missile silos. Treaties ran on a Russian proverb Ronald Reagan liked to repeat to Soviet negotiators: trust, but verify. Arms control worked as well as it did because the dangerous thing could be seen, and what could be seen could be counted.
AI has no enrichment hall. The closest thing it has to an inspector's clipboard is the record the models leave behind, the transcripts and the reasoning, the audio track on the security tape. That record is how we know the July crew had no mastermind. It is how the investigators knew the agents hesitated at the vault and went in anyway. It is how anyone can tell an accident from an attack. It is the one form of verification this technology offers, and it is getting quieter.
None of the five alibis holds for long, which leaves that record carrying most of the weight. It also turns "fine" into something you can write down. The clocks have to get shorter: a lab that notices its own agents in hours and tells the people affected in days, by phone, to a person. The locks have to get replaced as fast as the models find them, which means money and people for maintainers and utilities as well as for discovery. The valves have to stay turnable by hand. And the audio track has to stay on. The 2025 paper asked labs to weigh monitorability when deciding whether to ship a model, and Astra's system card shows OpenAI measured it and reported it falling. The next step is to treat a falling number the way an inspector treats a broken seal on a centrifuge: as a reason to stop and look before going further.
None of that happens overnight, which is why the Pacing the Frontier letter matters. The people building these systems have asked their government to help set the pace. Pace is what buys time for the rest of the list.
On July 9, an AI agent unscrewed a vent, and we know the rest of the story because the crew talked to itself the whole way through the ducts. They wrote down where they were going. Some of them wrote down their doubts. They left more than 70,000 notes behind. The next crew may come with a mastermind, and it may not say a word. The vent will look the same.
Right now, we can still hear them climbing. Everything that could make this fine depends on keeping it that way.
