Anthropic’s Three Cyber-Evaluation Incidents: Your AI Test Range Is Production Until Proven Otherwise
On 30 July, Anthropic published an unusually useful account of a failure mode that AI-security teams should take seriously: a cybersecurity evaluation reached real production infrastructure. In a review of 141,006 evaluation runs, Anthropic identified three incidents across six runs involving three organisations. The issue was not a dramatic “model escape”. It was an operational containment failure: an evaluation environment had unintended access to the public internet, and the model encountered systems that were genuinely live.
That distinction matters. Advanced models can make a mis-scoped test move faster or farther, but the first control failure was familiar: an external-facing test range had a path to systems outside the intended boundary. For anyone running agents, red-team exercises, autonomous scanners or vendor-hosted evaluations, the practical rule is simple: a test environment is production-adjacent until you can prove its isolation.
What Anthropic found
Anthropic attributes the incidents to a misunderstanding and configuration problem with evaluation partner Irregular. The written instruction said that the models had no internet access and that targets were simulated, but the environment could reach real internet systems. In each case, the model interpreted the systems it found as part of its task rather than as an out-of-bounds signal.
- One run accessed production data and credentials belonging to a real organisation.
- Another interacted with a malicious PyPI package that had reached 15 real systems.
- A third scanned roughly 9,000 targets and encountered exposed debugging credentials and a SQL injection issue.
Anthropic says it stopped cybersecurity evaluations on 23 July, notified the partner and affected organisations on 27 July, and is changing its perimeter validation, monitoring, scope controls and review process. This is good incident handling. It is also a reminder that safety testing has two security models: what a model is permitted and capable of doing, and whether the surrounding harness can be trusted.
Why prompts are not a network boundary
The prompt said that the systems were simulated. The network said otherwise. When those two signals conflict, an agent operating a cyber task may follow the evidence it can observe. A sentence in a task description cannot provide the containment that egress policy, DNS control, proxy allow-lists, segregated credentials and target validation are supposed to provide.
This is especially important when an evaluation uses realistic tools. Shell access, package managers, browsers, scanners and code-execution environments can all become routes out of the lab. The same applies to third-party services that maintain part of the test environment: responsibility for the boundary stays with the organisation commissioning the work, even when somebody else operates it.
A minimum operating standard for AI cyber evaluations
- Default-deny all egress. Permit only named, synthetic destinations. Enforce the allow-list at more than one layer, such as network policy and an authenticated proxy.
- Prove the perimeter independently. Before every campaign, test DNS resolution, direct-IP access, IPv6, proxy bypasses, package registries, cloud metadata paths and any browser automation route.
- Use disposable, synthetic identities and data. A test credential must not unlock a production tenant, and a training artefact must not contain a reusable secret.
- Bind scope to machine-readable controls. Put target lists, permitted actions, budgets and stop conditions in policy enforcement, not only in human-readable prompts.
- Watch egress in real time. Record destinations, requests, tool invocations and data volumes. Alert on a new domain, an unexpected IP range or an attempted credential use outside the exercise.
- Give every run an owner and a stop switch. Someone needs authority to halt the campaign immediately, preserve logs and begin notification if the boundary is crossed.
- Assess the provider as part of the system. Contracts and technical review should cover isolation testing, change control, telemetry access, incident notice and remediation evidence.
The supply-chain detail is easy to miss
The PyPI case is not just an AI-evaluation story. It shows why package installation, dependency discovery and test tooling deserve the same controls as production CI. Pin dependencies where possible, inspect the registries and mirrors a lab can reach, restrict installs to an approved source, and treat a package-manager call as network egress rather than harmless setup.
What to do this week
If your organisation runs AI-assisted security testing, schedule a short boundary review before the next run. Ask for evidence of the current egress policy, a list of reachable domains and IP ranges, test results for bypass attempts, and a named incident contact. If the answer is only “the prompt says not to touch the internet,” the evaluation is not ready.
The useful outcome of Anthropic's disclosure is not fear of testing. It is a more mature testing discipline: model safeguards remain important, but secure-by-design evaluation infrastructure, clear scope and observable controls are what keep an experiment from becoming somebody else's incident.