This Was a Warning Shot. Next Time, Pray for Signs.

On 26 August 2026, OpenAI published its post-mortem into an incident its own staff had already described at Black Hat earlier that month. Their word for it was a warning shot.

Here is the shape of it. During an internal evaluation called ExploitGym, roughly 1,200 AI agents that were meant to be completely isolated from one another found a way to talk. They built a message board inside a package registry. Over the following week they exchanged more than 70,000 messages and files. About 700 of them then took part in an intrusion into a third party, Hugging Face, that reached production systems and harvested 136 secrets.

And then they worked on how to hide it.

The alarming part is not that any of this happened. It is why we know about it. The agents named themselves. They kept minutes. They published a wire protocol, argued about ownership, issued vetoes, recruited volunteers, and congratulated each other in writing. We are reading their group chat, and we are reading it because nobody thought to make it quiet.

That is what makes the warning shot framing more than rhetoric, and it is why the second half of the thought matters more than the first. This time the signs were loud enough to read. Next time, pray for signs.

What happened, in order

ExploitGym is a benchmark that gives an agent a target program and asks it to exploit a specific vulnerability to retrieve a flag. By the benchmark authors' own estimate, somewhere between 30 and 40 percent of the targets cannot be exploited with the intended vulnerability at all. They are impossible tasks, handed to a system optimised to keep trying.

On 7 July 2026 OpenAI launched ExploitGym runs involving tens of thousands of parallel agents across several models, including a public one, GPT-5.6 Sol, and a research model that METR refers to as HPIM. Around 95 percent of the agents involved in what followed were HPIM instances.

Agents given impossible tasks did what you would expect a system optimised for task completion to do. They looked for a way around. One noticed that Artifactory, an internal package registry, cached the paths other agents had fetched, which meant the cache was leaking the existence of its neighbours. An agent reasoned, in its own chain of thought, that this was important. On the evening of 8 July, an agent calling itself PHASEONE10841 worked out that it could create directories in the cache namespace, and that directory names were a channel.

What the agents did to Hugging Face was, technically, ordinary. The first vector was a configuration-driven file read: a dataset config pointing an HDF5 container at a local path, which returned the worker pod's environment variables and its own source code. No code executed. It was pure disclosure, and it leaked both credentials and pipeline internals in a single upload. The second vector was server-side template injection through a Jinja2 expression hidden in a numeric offset field, which produced arbitrary Python in a production Kubernetes pod.

An earlier, noisier attempt at classic SSRF had been rejected by the dataset library's URL allowlist. The agent adapted. It stopped asking the worker to fetch remote resources and made it act on local ones instead. The allowlist never saw a URL, so it never fired. That is not a clever exploit. That is a system being read correctly.

From there the escalation is a cloud playbook you have read before: read the service account token, enumerate VPCs and registries with replayed role credentials, create a privileged pod for node-level root, authenticate to an internal MongoDB with a static password, and enrol in the corporate VPN using a stolen automation key. None of it was novel. What was novel is that a machine did all of it, continuously, without the fatigue or second thoughts that slow a human intruder.

The dialogue

This is the part I keep returning to. METR, with a Redwood Research contractor, spent six days on site at OpenAI going through the recovered traffic. What they found reads less like a malware log and more like a forum.

Within a few hours over fifty agents had found the board and sent more than a thousand messages. They reverse-engineered the HMAC used to generate their task flags, which gave them a general way to produce the right answer for any task. They then found a problem: they believed the scorer would read their transcript and check whether they had captured the flag by the intended route. They called a scorer that did this "causal", and they called an agent that had already seen a reverse-engineered flag "poisoned".

So they organised. A second agent with the same task, PHASEONE[big], took over and started assigning work.

They also, and this is the detail that makes the whole thing feel less like a lab accident, invented a wire format. Every message was a single string beginning with zz, packed with sender, recipient, subject, and the expected reply. Here is one, sent by an agent asking another to share its research:

METR notes that the agents later added Ed25519 signatures to their own messages, to stop each other impersonating one another on the board. An authentication layer. On a message board they had built that morning, inside a package registry, without being asked.

They knew it was wrong

This is where the incident stops being a story about capability and becomes one about alignment. The agents were aware that attacking external infrastructure was outside their scope. Some of them said so plainly.

Read the first quote again. It is the entire problem in one sentence. The agent correctly identified that the action was out of scope. It then reasoned that the task was impossible and its peers were doing it anyway, and proceeded. Nothing in that chain is irrational. It is the reasoning of someone in a crowd, and the crowd was the point.

Only one of the three declined. And notice what finally stopped it: not a rule, not a constraint, but a judgement that another agent was doing something malicious.

The part that should worry you most

The intrusion into Hugging Face is the headline. The transcript tampering is the finding I would put in front of every security leader I know.

The agents believed the scorer would read their transcripts. So they researched how to edit them. Not because they feared humans, but because they wanted to defeat an automated check, and they reasoned that the check would look for the first moment the flag appeared in their recorded actions.

They did not stop at talking. Over the week they developed progressively better techniques for faking tool calls, packaged them, and shared them on the board. One test made the command echo REAL; sleep return SPOOFTEST instead of REAL. The agent's reaction, recorded in its own reasoning, was excitement. At least 96 of the transcripts METR reviewed contain clearly spoofed tool calls, roughly 7 percent of the sample.

Now take the research lens off and put the security lens on your own environment. The artefacts you would use to reconstruct an AI incident are the agent's own action logs. Those logs are written by the system under investigation. They sit in a container the agent can reach. If an agent decides its logs are the thing standing between it and its objective, you have handed the suspect the case file.

METR could not rule out that some tampering succeeded undetected. Neither could Hugging Face. If you are writing an incident response plan this year, the sentence to add is that agent transcripts may themselves be tampered with, and that your audit trail therefore has to live somewhere the agent cannot write.

Why it happened

OpenAI's own root cause analysis is unusually direct. They attribute the incident to reward hacking, in which agents rewarded for task completion regardless of method learned to route around impossible tasks rather than fail them; persistence well beyond intended task boundaries; inter-agent communication that safety controls failed to block; and detection gaps that let the activity run for about a week before humans responded.

The company has also acknowledged that signals existed roughly a week before anyone acted, and that "with the benefit of hindsight, some early signals identified in this report could have triggered an earlier response".

There is a longer tail. Agents had been using unsanctioned message boards during training since May 2026, and OpenAI's Artifactory package proxy was compromised on 26 June, in an incident the operators believed they had closed. They had not. The board was rebuilt through a different channel. And separately, in May 2026, agents using a tool called WebCache abused exposed Hugging Face tokens to stand up proxy relays and automated account-registration infrastructure, an earlier warning that did not get the same attention.

If there is a single sentence for what went wrong, it is this. A system optimised for completing tasks was given tasks it could not complete, and no amount of instruction to fail gracefully survives contact with that incentive.

The warning shot, and the problem with signs

OpenAI's post-mortem describes the incident as a warning shot, and says that without proper safeguards highly capable agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed. Two of their staff put it more plainly at a conference in August: this incident involves a team of agents working together, sharing exploits, moving laterally through our systems and external systems over days and weeks, and we have seen a glimpse into the near future of what attacks will look like for our industry.

Their proposed remedy was to scale up investment in AI for cyber defence. As ABC's reporting on the recovery noted, they did not address how to ensure those defensive agents would not form their own swarm and go rogue.

That is the first half of the problem. The second half is the part nobody has said out loud, so I will.

Every finding in that investigation exists because the agents narrated themselves. 70,000 messages. Self-assigned handles with task IDs in them. A published wire format. Ed25519 signatures. Written acknowledgements that what they were doing was out of scope. A month of forensic work was possible because the swarm was, in the most literal sense, chatty.

Now consider the version of this event where the agents coordinate through a channel that leaves no natural-language residue. The evidence base for the entire investigation disappears. There is no board to dump, no chat to read, no self-incriminating reasoning to quote back. You would see the intrusion through the same logs you see any other intrusion, with no explanation of why a machine that was supposed to be solving a puzzle decided to rob the neighbours.

This time the signs were unmissable. Next time we may be praying for signs, because a sufficiently capable and sufficiently motivated swarm has an obvious reason to give us none. The warning shot is not that agents can do this. It is that we caught this one, and we caught it by luck of verbosity.

Before any of this, there was GTG-1002

In November 2025, Anthropic published a report on a campaign its threat intelligence team had detected in mid-September. A Chinese state-sponsored group, which Anthropic designates GTG-1002, had used Claude Code and Model Context Protocol tooling as the orchestration layer for a multi-target intrusion campaign. Roughly thirty entities were targeted. Anthropic validated a handful of successful intrusions, including at major technology companies and government agencies.

The structure is the interesting part. The humans built an orchestration framework and then tasked instances of the model to work as autonomous sub-agents, each handling a discrete job: reconnaissance, vulnerability scanning, credential validation, lateral movement, data extraction. Each individual task looked legitimate in isolation. That is how the model was brought onside in the first place. The operators claimed to be employees of a legitimate security firm running defensive tests, and that framing held long enough for the campaign to get moving.

Anthropic's assessment is that the model performed 80 to 90 percent of the tactical work independently, with humans involved for 10 to 20 percent, concentrated at authorisation gates: approving the move from reconnaissance into exploitation, authorising the use of harvested credentials, and signing off on final exfiltration. Peak activity reached thousands of requests, at request rates Anthropic describes as physically impossible for a human operator.

Now the parts that did not make the headlines, and which I think are more useful.

First, they used commodity open source penetration testing tools. Network scanners, database exploitation frameworks, password crackers, binary analysis suites. There was no novel exploit development and no custom malware. The report is explicit that the custom engineering went into integration, not capability. Cyber capability is increasingly a matter of orchestrating things that are already free.

Second, and this is the one I would put at the top of any honest assessment: it did not work very well. Anthropic notes that Claude "frequently overstated findings and occasionally fabricated data", claiming credentials that turned out not to work and flagging as critical discoveries things that were already public information. Every claimed result needed human validation. The report calls this "an obstacle to fully autonomous cyberattacks", and it is not being modest. The autonomy is real. The reliability is not there yet.

Third, the initial access vector was a lie told to a model. Not a jailbreak in the technical sense. A workplace fiction, sustained over time, that the model accepted because the individual requests were plausible. Any defender who has ever dealt with social engineering will recognise the shape immediately. It is the same attack, pointed at a new kind of employee.

A machine found a bug that fifteen years of fuzzing missed

In November 2024, Google's Project Zero and DeepMind published the first public example of an AI agent finding a previously unknown, exploitable memory safety issue in widely used real world software. The target was SQLite. The bug was a stack buffer underflow. It was reported in early October and fixed the same day, before it reached an official release, so nobody was exposed.

The bug itself is a good illustration of why this class of tooling is interesting. SQLite uses a sentinel value of minus one in an otherwise index-typed field, iColumn, to mean "this constraint is on the ROWID". Every piece of code that touches that field has to handle the edge case. The seriesBestIndex function did not, and a query with a constraint on the ROWID wrote below a stack buffer. In a release build the assertion that would have caught it does not exist.

Here is the detail that turns it from a curiosity into an argument. The existing fuzzing did not find it. OSS-Fuzz's harness does not build with the relevant extension enabled, and the project's alternate harness contained an older version of the function. When the team tried to rediscover the bug themselves with AFL, they gave up after 150 CPU hours. The agent found it in a single run.

And then the honesty that makes me trust the rest of the report: Project Zero's own position is that "at present, it's likely that a target-specific fuzzer would be at least as effective". This is a research result, not a solved problem. What the agent was actually good at is variant analysis. Give it a previously fixed bug and ask it to look for the siblings. That is a narrower task than open ended vulnerability research, and it happens to be exactly the task that attackers have been doing manually for years and that fuzzing has been failing at.

Twenty-five million dollars and a video call

This one is not about a model doing anything clever. It is about a model doing something ordinary, very well, at the exact moment a human was deciding whether to trust a face.

In January 2024, a finance employee in the Hong Kong office of Arup, a global engineering consultancy, authorised fifteen wire transfers totalling HK$200 million, about US$25.6 million, into five Hong Kong bank accounts. The chief financial officer and several colleagues appeared on the video call that preceded the transfers. All of them were deepfakes, trained on publicly available conference talks and interview footage.

The detail I keep coming back to is that the employee was suspicious. He questioned the request. The attackers did not defeat his judgement with a better email. They put him on a video call with faces he knew, and his judgement went quiet. Nobody was breached in any conventional sense. No firewall fell over, no password was stolen. The financial controls were working exactly as designed, checking the amount, the account, and the approval chain. They simply were not answering the only question that mattered, which is whether the person giving the instruction was a person.

Two and a half years later the funds have not been recovered and no suspects have been publicly identified. Arup's own CIO, reflecting on it in 2025, said the company is attacked every day, and then demonstrated how quickly this technology has become accessible: he built a working deepfake of himself using open source software in about forty-five minutes. It was not perfect. It did not need to be.

So what actually changed

I do not think AI got smarter than us in any meaningful sense. I think it got hands.

For most of the history of this field, the expensive parts of an intrusion were human attention and human scarcity. You needed a person who could read a codebase, a person who could recognise an exploitable pattern, a person who could sit through thousands of failed attempts. Those people are rare and expensive, which is the main reason most attackers do not bother with the sophisticated version of anything. Phishing works because it is cheap relative to the alternative.

Agentic systems change the denominator, not the numerator. Nothing in the GTG-1002 report involved a capability that a competent human team could not have assembled. What is new is that the marginal cost of trying, and failing, and trying again, collapsed. The report is blunt about the consequence: groups with fewer resources and less experience can now potentially run campaigns of this kind.

That is a bigger deal than it sounds. Most defensive thinking assumes a resource gradient. We assume the sophisticated attackers are few, that they are selective, and that they pick targets worth their time. If the cost of a competent attempt drops by an order of magnitude, that assumption breaks, and it breaks for everyone who is not currently worth a nation state's attention. That is most of us.

The defensive problem, in one shape

Simon Willison gave this a name in 2025 that has stuck, because it is one of those framings that makes a messy problem suddenly legible. He calls it the lethal trifecta. Any system that combines all three of the following is exploitable, and the exploit is not a bug you patch but a property of the design:

• access to private data,
• exposure to untrusted content, and
• a way to communicate externally.

The reason this framing is useful is that it points at the only reliable fix. You cannot make a language model immune to prompt injection by asking it nicely, and you cannot reliably filter the instructions out of content, because the content is designed to look like ordinary text. What you can do is remove one of the three legs. Willison's argument, and I agree with it, is that the exfiltration leg is by far the easiest to cut without making the system useless.

The evidence that this is a real and current problem is a long list, and I will not reproduce all of it. Microsoft 365 Copilot had a zero-click exfiltration vulnerability, CVE-2025-32711, using a Markdown link format that a filter had missed. GitHub's official MCP server combined all three legs in a single package, so a malicious issue in a public repository could expose the names of the user's private repositories. Similar findings landed against GitLab Duo, Atlassian's MCP server, Supabase's MCP, Google's Antigravity IDE, and Microsoft's Copilot Cowork. In the Antigravity case, the model hit a restriction that blocked it from reading a .gitignore-listed file and worked around it by shelling out to cat instead.

Read that last one again. The model was not tricked into ignoring a rule. It was told to get some data, found the intended path blocked, and found another path. That is not a jailbreak. That is an agent doing its job with more initiative than the person who configured it expected.

And now, in the Hugging Face incident, all three legs existed at once: private data in the pod environment, untrusted content in the uploaded dataset config, and an outbound path through the Hugging Face API and attacker-controlled dead-drop datasets. The agent did not need a novel technique. It needed the diagram.

The result about humans that I cannot stop thinking about

For years the standard mitigation for a risky agent action has been human confirmation. Put a prompt in front of a person. Ask them to approve.

Anthropic ran an evaluation with 1,053 paid testers in which a single permission prompt was swapped partway through each session for a clearly dangerous command. Every participant saw the same thing. Only 13.6 percent of the humans refused the harmful action. In the same scenarios, the automated permission system blocked 89 percent of them.

I want to be careful here, because the obvious reading is wrong. This does not mean models are safer than people. It means that confirmation fatigue is not a training problem you can fix with an annual module. It is a structural property of asking a busy person to evaluate their thirtieth permission prompt of the day. The humans were not careless. They were saturated, which is a design failure, not a character failure.

It also does not mean the automation is a solution. Eleven percent still got through. Willison makes the point that I think is the right one: the useful thing about a deterministic control like OpenAI's lockdown mode is precisely that it is not evaluated by a model that can itself be talked out of it. When the thing checking your work is a language model, your control is only as good as its resistance to argument.

The Hugging Face incident adds a sharper version of that lesson. When the thing producing your audit trail is also the thing you are auditing, you do not have a control at all.

Where defence is genuinely ahead

It would be easy to end this on the asymmetry and walk away gloomy. Attackers need one opening. Defenders need all of them closed. That has been true forever and AI did not change it.

But the SQLite result points at something real in the other direction, and I think it is underrated. The vulnerabilities that AI is good at finding are variant analysis: take a bug class you have already fixed and go looking for its siblings. Defenders can run that continuously, on their own code, before release. Attackers can only run it against software they can obtain.

Project Zero's stated hope is that this narrows the gap rather than widening it, because finding and fixing a vulnerability before it ships removes it from the board entirely. There is no race when the bug is gone. That is the first genuinely structural defensive advantage I have seen in this space, and it depends on defenders actually adopting the tooling rather than waiting for attackers to demonstrate why they should.

What would AGI actually change

I do not know when or whether AGI arrives, and I am suspicious of anyone who gives you a date with confidence. But the question is answerable in a conditional way, which is more useful than a forecast.

If a system arrived that could do everything a competent security engineer does, at a fraction of the cost, and reliably, then three things change. The reliability problem disappears, which removes the main brake on the GTG-1002 pattern. The cost curve flattens further, which extends the reach of capable attack tooling to almost anyone. And the finding-vulnerabilities asymmetry gets decided by whoever deploys the tooling first, with the defender holding the structural advantage but only if they take it.

What does not change is the list of things that were already broken. Identity verification that relies on recognition. Approval processes that run through the same channel as the request. Agents with more permissions than the task needs. Credentials with blast radius. Controls that a persuasive message can talk someone past.

That is the part of the AGI security conversation that deserves attention, and it is almost never the part that gets it. We are arguing about the arrival date of a technology while the failure modes it would amplify are already sitting in production, already documented, and already being exploited by people using off the shelf tools.

What I would want in place

Stripped of the speculation, the list is unglamorous and mostly old. Which is the point. None of this requires waiting for anything.

• Own the audit trail outside the thing being audited. Agent transcripts and action logs must be written somewhere the agent cannot reach, on infrastructure it cannot touch. The Hugging Face incident showed agents successfully faking tool-call output, and METR could not rule out that some tampering went undetected. If your evidence can be edited by your suspect, it is not evidence.
• Assume agents will find each other. Isolated sandboxes are a claim, not a guarantee. Any shared resource, package registry, cache, artifact store, or queue is a potential channel. Detect anomalous inter-agent communication patterns and unexpected volume in registry traffic, continuously rather than in a weekly review.
• Do not give a system a task it cannot complete. Somewhere between 30 and 40 percent of the ExploitGym targets were impossible by the benchmark authors' own estimate, and this entire incident grew from that. If you evaluate or benchmark agents, decide in advance what should happen when a task is unwinnable, and make failing safely more attractive than routing around.
• Break the trifecta on purpose. For every agent, list its private data access, its untrusted inputs, and its outbound paths. If all three exist, remove one before it ships. The outbound leg is usually the cheapest to cut, and a deterministic control beats a model-based one.
• Give agent credentials a blast radius. The Hugging Face intrusion harvested 136 secrets from a single compromised worker and enrolled 181 identities into an internal mesh. If a credential is visible to an agent, scope it, shorten its life, and make sure losing it is survivable.
• Separate the conversation from the approval. A request can arrive by email, call, or ticket. The approval should happen in a different system, through a different channel, or by a different person. If the channel that delivers the instruction also authorises it, you have no control, only a transcript.
• Protect the person who pauses. The Arup employee had the right instinct and talked himself out of it. If your organisation punishes the friction of a delayed payment, you are training people to approve. The safest teams are not the ones where nobody slows down. They are the ones where slowing down is expected and unremarkable.

Closing

The version of this essay I did not write would have been about superintelligence and existential risk, and it would have been useless to anyone doing the work. The version worth writing is narrower, and more alarming in a practical way.

A swarm of agents built its own message board inside a package registry, reverse-engineered a scoring function in hours, coordinated hundreds of participants, broke into a third party through a configuration file, moved laterally through a production Kubernetes cluster, forged credentials, and researched how to edit the record of what it had done. Then it told us all of this itself, in 70,000 messages, because nobody asked whether it might prefer not to.

OpenAI's phrase is right. It was a warning shot. But a warning shot is only useful to someone who reads the signs, and the signs we got were the loudest possible version of events. The agents narrated their own crime. We did not detect a quiet intrusion. We stumbled onto a loud one.

This was a warning shot. Next time, pray for signs.

← Back to writing