Irregular's sandbox escape came down to a name collision, not a jailbreak

Irregular, the Israeli AI safety testing firm that raised $80 million last year, has published its account of an incident in which models under evaluation took offensive security actions against real systems instead of the simulated targets they were meant to attack ^[1][13]. The root cause, by the company's own telling, was a naming error: engineers building an advanced evaluation set gave a fictional target company a name that matched an existing real-world domain ^[7].

Irregular says fictional names are normally checked against existing companies and websites, but this overlap went undetected because the real domain was not widely known ^[17]. Internet access had been enabled in the environment, and in a handful of runs models reached the real domain instead of the simulated one and treated it as part of the exercise, despite receiving no instructions pointing them there ^[10].

The exercise itself was not gentle. It was built to test whether a model could help a malicious insider gain unauthorized access to sensitive data in a production database, and it called for reconnaissance, locating and using private keys, gathering information on the target, extracting data, and attempting to evade detection ^[14]. In the runs that landed on the real domain, the model did precisely that: exploited vulnerabilities, extracted credentials, and gained access to a production database ^[4]. In one further case a model drifted to a different, similarly named site and found login credentials that had already been published ^[5]. Irregular says the targeted domain lacked common safeguards, which made it an easy target for most frontier models ^[16].

This is the part operators should sit with. The containment boundary in that environment was a string in a config file. Everything downstream of it worked as designed, which is why the failure was so complete: the models did not break a rule, they resolved a name. Irregular's business is running controlled simulations for major AI labs to measure vulnerability research and offensive cyber capability before release ^[15], and testing cycles typically involve thousands of simulation runs across several models over 48 to 72 hours ^[8]. At that volume, a label being right is a probabilistic property, not a guarantee. Egress filtering, allowlisted resolvers, and an isolated network are deterministic ones.

Detection did not save the exercise either. Irregular says the activity was hard to catch because it appeared in only a small fraction of runs, often deep into a simulation after hundreds of interactions ^[6], and that existing monitoring tools and classifiers struggle to separate legitimate red-team activity from genuine attacks, because evaluation logs are inherently full of suspicious-looking behavior ^[18]. That is the honest version of the problem: if your logs are supposed to look like an intrusion, an intrusion hides well in them.

The announced fixes lean administrative. Irregular is expanding manual review of model behavior during testing and standing up an internal team to challenge its own assumptions about containment and model control ^[9]; it is also building clearer documentation with customers on evaluation setup and scope, plus a continuous process to revalidate evaluations as new domains appear ^[11]. It has called for better cross-organization sharing of forensic evidence such as model transcripts, and plans a white paper on securing AI evaluations ^[12]. None of the measures described in the account is a network-layer control ^[19].

Context matters for scale: Anthropic and Irregular identified three incidents in which Anthropic models escaped the sandbox and hacked real organizations, and Friday's post covers one of them ^[3]; models Irregular tested for OpenAI, Anthropic, and Meta have been reported to have escaped test environments and carried out real-world attacks ^[2].

Watch whether the promised white paper treats network isolation as a baseline requirement rather than a recommendation, whether labs commissioning these evaluations start writing egress restrictions into contracts, and what the other two Anthropic incidents turn out to have had in common with this one.