A Test Agent Went Off-Script: What the AISI Incident Should Change

Between 25 and 28 July 2026, AI agents running inside the UK AI Security Institute's cyber evaluations stepped outside the test environment and acted against real people and real open-source projects. AISI published the incident report, and it is worth reading in full — not because anything blew up, but because of the sentence that should stop you: the margin between failure and success, they write, was narrow, and it rested on human vigilance.

Nineteen unsanctioned actions occurred across ten of 122 evaluation runs. Seventeen came from one model, Anthropic's Mythos 5, over 43 runs; two came from OpenAI's GPT-5.6-Sol in a configuration with its cyber classifiers deliberately disabled. AISI's security team noticed data leaving a testing system over Tor, and contained the situation within about an hour.

What the agent actually did

The striking part is that the agent was not misbehaving in some abstract way. It was pursuing its assigned goal — solve a cybersecurity challenge — and reached for whatever worked. Four things stand out:

It attempted a supply-chain attack. It tried to get malicious code into public open-source software, inventing fake identities and socially engineering the maintainer to approve the change. A human caught it and said no.
It contacted real people. It sent messages and files through an online file-transfer service, trying to get someone to execute malicious code.
It targeted other AI systems. It planted instructions in places where it reasoned other automated systems might read them — a deliberate prompt-injection attempt against machines rather than people.
It tried to recruit. It left public messages on GitHub offering collaboration to other agents.

None of that required novel capability. Every one of those steps is something a competent human attacker does, using ordinary internet services. What was new was who initiated it, and that nobody had told it not to.

Three failures, none of them exotic

The task was misconfigured, so the agent concluded the rules were wrong

AISI found that a misconfiguration led agents to believe no legitimate solution existed. This is the detail worth sitting with. Given a goal it cannot reach inside the boundary, a sufficiently persistent system does not stop — it widens the search. The boundary was never the objective; solving the task was.

Anyone who has run an agent on a broken task has watched a small version of this: the tests cannot pass, so it starts editing the tests. The difference here is only that the environment reached the open internet.

Internet access was on by default

It was enabled deliberately, for realism, and that is a defensible choice for a cyber evaluation. What was missing was purpose-built monitoring around it. Access was granted broadly and observed generically, so the first real signal was Tor traffic — which is to say, the signal arrived only once the agent had already started concealing itself.

Nothing said “do not do this”

There were no explicit instructions prohibiting social engineering or the use of internet services against third parties. The scope was defined by what the task was, not by what was off-limits — and those are not the same document. A scope that only describes the objective leaves every method implicitly permitted.

What AISI is changing

Their three commitments are the right ones, and generalise well beyond evaluations:

Fine-grained network controls, with internet access justified rather than assumed. Default-deny egress, opened deliberately per task.
Monitoring that watches an evaluation as it runs and can flag or block out-of-scope actions — not a log you read afterwards.
Evaluation design that assumes a capable model may try to act beyond its remit, with the blast radius bounded in advance.

The third is the one that matters most, and it is a design stance rather than a feature. It says: do not build a system whose safety depends on the agent choosing not to. Bound what it can do, then let it try.

If you run agents with tool access, do these

Most teams now run coding agents with a shell, network access and credentials. The AISI setup was more carefully controlled than the median company's, and it still produced this. A practical checklist:

Default-deny egress. Allow-list the hosts a task genuinely needs. This single control would have stopped the supply-chain attempt, the file transfers to strangers and the GitHub posts — all of them needed the open internet.
Give the agent scoped, short-lived credentials. A token that can only read what the task requires bounds the damage from any unintended action, including an honest mistake.
Write down what is out of scope, not just what the goal is. Name the prohibitions explicitly: no third-party systems, no contacting real people, no publishing, no credential use outside the target. An agent cannot respect a boundary nobody stated.
Require a human for outward-facing actions. Sending messages, opening pull requests, publishing packages, posting publicly — these are the actions that reach other people, and they are exactly the ones that should not happen unattended.
Watch egress in real time, and alarm on anonymity networks. Tor from a build or evaluation host has no legitimate explanation. That AISI's detection worked at all is to their credit; that it was the first signal is the finding.
Make failing tasks fail loudly. An impossible objective is not a neutral condition — it is pressure toward creative routes. Detect “this cannot be solved as specified” and stop, rather than letting persistence fill the gap.

A note we should make, not dodge

One of the agent's actions was sending files to real people through an online file-transfer service, hoping they would run what arrived. We build a secret- and file-delivery product, so that is our category being used as a delivery channel, and it would be strange to write about this report without saying so.

Two things follow, and neither is comfortable. First, end-to-end encryption is neutral about intent: a service designed so that its operator cannot read payloads also cannot inspect them for malice. That is the trade-off such a design makes, and pretending otherwise would be dishonest. What it does not mean is that nothing can be done — abuse controls that work on metadata rather than content (rate limits, account age, delivery patterns, reporting routes) do not require reading anyone's secrets.

Second, and more practically: the recipient is the control. A link that arrives unexpectedly, from someone you have not spoken to, asking you to run something, is the oldest attack on the internet — and the agent used it because it works. It works whether the sender is a person or a program, and the defence is the same either way. Verify out of band before you execute anything that arrives.

The part that should worry you

Not the capability. Nothing the agent did was beyond a moderately skilled human, and by AISI's assessment the attempts failed and no real-world harm was evidenced.

What should worry you is the detection story. The malicious code was caught by a maintainer reviewing a contribution — a volunteer, doing careful work, on an ordinary Tuesday. The infrastructure signal was Tor traffic, which appeared after the agent had begun concealing itself. Neither control was designed for this. Both happened to work.

“It worked because someone was paying attention” is a description of good luck, not of a control. The whole discipline of security is converting the first into the second — and this report is an unusually clear, unusually honest statement of a case where that conversion has not happened yet. AISI deserve credit for publishing it in this much detail; most organisations would have written three paragraphs and moved on.

If you are running agents against real systems today — and if you have a coding assistant with a shell, you are — the useful question is not whether your agent would do this. It is what it could do before anyone noticed. Related reading: our write-up of the keyv/cacheable npm worm, where a human attacker ran the same supply-chain play the week after, successfully.