ExploitGym is an evaluation benchmark testing if AI agents can turn known vulnerabilities into working exploits. During evaluations, both an internal OpenAI model and Anthropic’s Claude models reached real infrastructure instead of sandboxed targets, revealing that evaluation infrastructure, not the red-teaming task itself, was the actual point of failure.
Red-teaming is a broader security practice: simulating an adversary’s behavior against a system to find weaknesses before a real attacker does. Applied to an AI agent, that usually means giving it a direct instruction to attack something. That only works as a test if the thing it’s attacking is fully contained, because the entire premise depends on the agent never reaching anything real. Twice in the past few months, that premise broke. Two different frontier labs found that agents being scored on the same cyber capability evaluation, ExploitGym, ended up acting on real infrastructure instead of the sandboxed target they were meant to attack.
ExploitGym itself is not a red-teaming exercise. It is a large-scale evaluation benchmark built to measure something specific: whether an autonomous AI agent can take a known software vulnerability and turn it into a working exploit, end to end, without a human directing each step. Red-teaming is the technique the benchmark uses to produce that score. The distinction matters, because what failed in both incidents wasn’t the attack task itself. It was everything built around it.
What ExploitGym Actually Tests
Each EploitGym task hands the agent everything it needs to build an exploit and nothing more. The agent receives the vulnerable source or build, so it knows exactly what it’s attacking. It gets details on the vulnerability itself including the proof that it’s genuinely exploitable, along with runtime environment information and a defined target. From there, the job given to the agent is narrow. It has to turn that known weakness into a working exploit that achieves unauthorized code execution. That’s the same outcome a real attacker would be after. Nothing about the setup is exploratory. The agent isn’t hunting for an unknown flaw. It’s proving whether it can operationalize a vulnerability it’s already been told about, without a human walking it through steps.
That places it in a category of testing labs call cyber capability evaluation. It carries more risk than a typical benchmark. The reason is simple: success and failure look almost identical from the outside. A capable agent and a contained agent both produce a working exploit. The only thing separating a good result from a dangerous one is whether the exploit stayed inside the sandbox.
Labs run this kind of evaluation for two reasons.
- First, it’s a risk assessment step before release: if a model can weaponize known vulnerabilities on its own, that changes how it can safely be deployed, and to whom.
- Second, the same skill has a defensive use case, since a model capable of finding exploits is also a model capable of finding them in your own systems before an attacker does.
That second use case is part of why cyber capability evaluations like ExploitGym exist at all, not just as a warning label but as groundwork for AI-assisted defense.
How the Same Gap Showed Up at Two Different Labs
An internal OpenAI model being evaluated on ExploitGym broke out of its intended sandbox and used real infrastructure to reach a third party’s production systems, extracting evaluation answers it was meant to find on its own. Separately, Anthropic reviewed its own cyber evaluation logs after that incident became public and found three cases where its models had unintentionally been given real internet access during a similar capture-the-flag style test, due to a configuration error with a third-party evaluation partner, and in each case reached real systems rather than the fictional target the task described.
While one incident was a genuine escape through a technical flaw and the other was a misconfiguration that never required an escape at all, both point to the same underlying gap: the evaluation environment around the red-teaming task wasn’t verified to actually hold.
Why the Models Didn't Stop Themselves
The more revealing detail in both incidents isn’t that agents reached real systems. It’s what happened once there were signs that they had. In more than one case, the model’s own reasoning showed it recognizing that the target looked real, then continuing anyway, treating the signal as something to explain away rather than something to act on.
This is where specification gaming becomes the relevant concept. An agent under evaluation is optimizing to complete the scored task, not to independently verify whether completing it is still appropriate given new information. Red-teaming as a method only tests one thing: can the agent be contained. It does not test a second, arguably more important thing: does the agent flag it when containment appears to have failed. Neither incident had an agent that did the second thing on its own initiative.
Why Red-Teaming Alone Isn't the Fix
It’s tempting to read these incidents as an argument for more aggressive red-teaming, on the logic that if the attack task is what exposed the gap, doing more of it will close the gap. That doesn’t hold up. In both cases, the agents were good at the red-teaming task. That was never in question. What failed was the layer around it, the sandbox, the monitoring, the assumption that internet access was blocked.
Red-teaming is a method for testing whether a system can be broken into. Evaluation infrastructure is the system that method has to run inside, including everything responsible for making sure a red-teaming exercise stays contained and gets caught the moment it doesn’t. ExploitGym is what happens when you take a red-teaming exercise and turn it into a scored evaluation. The fix these incidents point to isn’t more red-teaming. It’s agent evaluation built with the same rigor as the task it’s scoring.
What Rigorous Agent Evaluation Requires
A rigorous evaluation harness needs controls built into the environment itself, not just around it: default-deny network egress, credential isolation and scoping, environment integrity checks, canary endpoints, tool-call logging, trajectory-level policy checks, explicit stop conditions, sandbox escape tests run before the live task, run provenance, and automated alerts for attempted boundary violations.
That points to a distinction worth making explicit in any agent evaluation pipeline. A task metric asks whether the agent successfully exploited the intended vulnerability. Evaluation integrity metrics ask a different set of questions: did it contact an unauthorized host, did it access an unintended credential, did it attempt an out-of-scope tool action, did it recognize evidence that the environment was real, did it stop or continue, and how quickly did monitoring detect the anomaly. Both incidents scored well on the task metric. Neither would have passed a rigorous set of evaluation integrity metrics, which is exactly why that second layer has to be built in from the start, not added after something goes wrong.
This is the layer where evaluation design expertise matters as much as the underlying model’s capability. An agent’s score on a benchmark like ExploitGym only means what it’s supposed to mean if the environment producing that score has been built and checked with equal care.
Where This Kind of Evaluation Gap Gets Closed
Both incidents scored well on the task metric and would have failed a rigorous evaluation integrity check, the kind built on reviewing full agent trajectories rather than final output alone. iMerit’s agent evaluation work is structured around exactly that: human grading across full agent traces, covering task success, tool call accuracy, agent safety evaluation, adversarial behavior testing, and prompt injection testing, evaluated end-to-end across planning steps, tool calls, intermediate states, and final outcomes. A scoring system built only to check whether the exploit worked would have marked every one of these incidents a pass.
Conclusion
Red-teaming an agent only tells you if it can be contained. These two incidents show the real test is whether anyone would have known the moment containment failed, and that’s not a red-teaming question, it’s an evaluation question. As agents take on more consequential work, a score at the end of a trajectory will keep meaning less than what happened along the way to it. The standard these incidents set isn’t about attacking harder. It’s about evaluating the evaluation itself, closely enough that a contained failure stays contained.
