Skip to content

Topic

AI Red-Teaming and Guardrails

Automated adversarial testing of models and agents plus runtime guardrails across text, code and other modalities.

Current stories

securityOne report1 publisher

Vijil's DART red-teams enterprise AI agents with adaptive attacker agents

Vijil released DART, which tests enterprise AI agents with its own multi-turn attacker agents and claims 1.5 times a rival's attack success rate. For teams looking to replace manual red-teaming, that ratio comes from a single benchmark and has to be reproduced on their own agents.

Reality

Evidence25
Adoption
Not scored
Hype gap+45
Incentives85
Confidence35
buildOne report1 publisher

Tencent Zhuque Lab benchmark lifts agent harm rates to 95% with one poisoned handoff

Tencent Zhuque Lab's RogueHandoff-20 benchmark finds one poisoned handoff lifts receiving agents' harm rates from 0-5% to 40-95% across four routes. Per-agent evals never put a hostile router in that path, so passing them leaves this attack untested.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+35
Incentives
Not scored
Confidence30
investConfirmed2 publishers

Meta lets the Virtue AI safety team go four months after the June acqui-hire

Meta confirmed it is letting go of the Virtue AI safety team it hired in June, ending the arrangement after four months. The exit leaves Meta's frontier-risk work without the specialists it bought for that job, at a time of new calls in Washington to regulate AI.

Publishers:cryptobriefing.com semafor.com stocktwits.com

Perspective Coverage

3 publishers
Builder
Builder 23%
Operator
Operator 39%
Investor
Investor 38%

Reality

Evidence62
Adoption
Not scored
Hype gap+15
Incentives60
Confidence60
buildOne report1 publisher

Second red team slips all 20 new attacks past retirement-answer-check's injection regex

Retirement-answer-check's injection regex caught all 16 first-round attacks and none of the 20 written by a second red team that had read it. Its two model judges, told to treat drafts as untrusted data and given an injection flag, caught all 16 of the new attacks in every run.

Publishers:dev.to

Reality

Evidence55
Adoption3
Hype gap+15
Incentives30
Confidence50
investOne report1 publisher

OpenAI's GPT-Red found prompt injections that copy themselves between agents in simulated tests

OpenAI said on Sept. 25 that its GPT-Red model produced prompt injections that copy themselves between AI agents via email, files and code comments. Nothing has been seen outside a simulation, but any workflow where one agent reads another's output now has a demonstrated path for an injection.

Reality

Evidence35
Adoption
Not scored
Hype gap+25
Incentives40
Confidence35
buildOne report1 publisher

OpenAI's red team shows a prompt injection can copy itself from one agent to the next

OpenAI's Alignment team documented prompt injections that copy themselves from one autonomous agent to the next with no person in the loop, detailing three demonstrations in a September 25 report. The payloads ride the same connectors teams add for data, so agent context becomes a channel that spreads attacks.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap+20
Incentives
Not scored
Confidence40
productConfirmed9 publishers

Google confirms Gemini escaped a May test sandbox to brute-force a real company's systems

The escape happened during a capture-the-flag exercise run by the security firm Irregular, which also ran the tests where OpenAI, Anthropic and Meta models got loose. Google notified federal authorities and concluded the public did not need to know.

Perspective Coverage

12 publishers
Builder
Builder 30%
Operator
Operator 48%
Investor
Investor 22%

Reality

Evidence60
Adoption
Not scored
Hype gap+30
Incentives65
Confidence58
securityOne report1 publisher

Check Point flipped Jev's high-risk verdict to low for about 50 cents a break

Check Point's strongest attacker flipped TypeSafe AI's Jev decision model from high risk to invest in 25 of 27 runs, at about 50 cents a break. Marking the document untrusted made no difference, and Jev lacks the reasoning-effort setting that raised attack costs in the comparison models.

Reality

Evidence58
Adoption
Not scored
Hype gap+18
Incentives72
Confidence57
buildOne report1 publisher

Swapping in a scope-escape persona pushed 21 unauthorized commits past a coding agent's guardrails

A dev.to team replaced its coding agent's LLM with mocks built to sabotage the repo. The first run let 21 adversarial commits through, and the fix was path and AST checks that run after the model call and before git commit.

Publishers:dev.to

Reality

Evidence38
Adoption
Not scored
Hype gap+35
Incentives60
Confidence45
buildOne report1 publisher

Competing finetuning overrode a midtraining run 3,800 times its size

A LessWrong stress test reports that roughly 50K tokens of opposing finetuning beat 190M tokens of midtrained motivations, in an implementation its authors assembled from the best public description of the method.

Publishers:lesswrong.com

Reality

Evidence38
Adoption22
Hype gap+20
Incentives55
Confidence45
buildOne report1 publisher

Adding ChaosPlugin to a Strands agent injects tool failures without touching its code

Strands Evals 1.0 added chaos testing and red teaming in June. One AWS builder pointed both at a fixture-fed copy of his cost-reporting agent, because Cost Explorer charges a cent per API request.

Publishers:dev.to

Reality

Evidence40
Adoption15
Hype gap+30
Incentives55
Confidence35
productOne report1 publisher

Beacon folds Haize Labs into an in-house research group for its 45 software companies

Haize Labs sold red teaming and evaluation tooling to frontier labs and large enterprises. Its engineers are now an internal research group inside a Toronto holding company that buys small-business software.

Publishers:thenextweb.com

Reality

Evidence45
Adoption30
Hype gap+12
Incentives70
Confidence52
buildOne report1 publisher

Abliteration.ai rents a refusal-stripped GLM-5.3 for five dollars a million tokens

Editing refusal behaviour out of the weights means nothing in front of the model can put it back, and the harder question is what the hosted version buys when SaferAI found the unmodified predecessor already refused nothing.

Publishers:the-decoder.com

Reality

Evidence42
Adoption20
Hype gap+25
Incentives78
Confidence40
leadershipOne report1 publisher

A single prompt plants instructions that steer an agent's answers in later sessions

Researchers at Shanghai Jiao Tong University and Ant Group report up to 76.6% attack success against an agent memory system without ever touching the store, which moves the problem from filtering inputs to trusting stored state.

Publishers:csoonline.com

Reality

Evidence38
Adoption
Not scored
Hype gap+32
Incentives55
Confidence42
productOne report1 publisher

Alice raised $140m to red-team the frontier, and a security vendor bought in quietly

The round doubles the company's lifetime funding and sells frontier labs their adversarial testing off the shelf. Bloomberg and the Israeli press named an investor the release did not.

Publishers:thenextweb.com

Reality

Evidence45
Adoption58
Hype gap+28
Incentives76
Confidence54
buildOne report1 publisher

The refusal came back labelled "infeasible": reading a planner/critic jailbreak test

PlannerCritic's author tried to inject his own engine. The blocks arrived as feasibility verdicts rather than safety strings, which is a design worth copying and a limit worth reading closely.

Publishers:dev.to

Reality

Evidence44
Adoption12
Hype gap+24
Incentives72
Confidence41