Skip to content

Benchmark

CyberGym

Benchmark that evaluates AI models on cybersecurity tasks such as source code review and vulnerability discovery or verification.

Known aliases

  • CyberGym-style harness
  • CyberGym-style tasks

Current stories

securityConfirmed6 publishers

Anthropic sends unreviewed AI bug reports with proofs of concept to open source maintainers

Anthropic expects more than 90% of the unreviewed vulnerability reports its new OSS Scanner sends to open source maintainers to be real. Opt-in projects get findings faster and take on the work of catching the errors, such as wrong severity ratings.

Perspective Coverage

6 publishers
Builder
Builder 52%
Operator
Operator 37%
Investor
Investor 11%

Reality

Evidence60
Adoption35
Hype gap+15
Incentives70
Confidence62
securityOne report1 publisher

Microsoft's AI-driven FORGE lab helped find 140 Windows CVEs in five months

Microsoft's FORGE lab helped discover Windows flaws assigned 140 CVEs from May to September 2026, 52 of them fixed in the September release. Microsoft says the pace now depends on how fast its response center can validate and fix what the AI agents report.

Publishers:microsoft.com

Reality

Evidence40
Adoption30
Hype gap+15
Incentives70
Confidence40
buildConfirmed4 publishers

Post-training alone took GLM-5.3 from 4.6 to 28.3 on Terminal-Bench 3.0

Z.ai says the base model did not change between GLM-5.2 and GLM-5.3, so the coding jump and the doubled exploitation score come out of the same post-training run. Security teams inherit the second half.

Perspective Coverage

4 publishers
Builder
Builder 49%
Operator
Operator 32%
Investor
Investor 19%

Reality

Evidence50
Adoption30
Hype gap+30
Incentives70
Confidence60
buildOne report1 publisher

GLM 5.3 adds no parameters, and its sharpest jump is in finding vulnerabilities

Zhipu says every gain in GLM 5.3 came from post-training on the same ~744B MoE it shipped as GLM 5.2. The largest claimed leap, in vulnerability discovery, is also the least quantified.

Publishers:dev.to

Reality

Evidence25
Adoption10
Hype gap+35
Incentives70
Confidence35
productOne report1 publisher

AI vulnerability scanners flag 41% to 99% of safe code in AWS's Deception Benchmark

AWS's Deception Benchmark found AI vulnerability scanners catch up to 95% of real bugs but flag 41% to 99% of safe code. Its samples were built to fool models, so teams still need their own false-alarm count before sizing the triage work.

Publishers:devops.com

Reality

Evidence45
Adoption
Not scored
Hype gap+20
Incentives
Not scored
Confidence40
buildConfirmed5 publishers

GLM-5.3 keeps GLM-5.2's base model and claims 50% more on coding: plan for shorter eval cycles

Z.ai says every gain in GLM-5.3 came from post-training on an unchanged base. If that holds, refresh cadence for self-hosted weights is set by RL runs, not pretraining runs.

Perspective Coverage

5 publishers
Builder
Builder 58%
Operator
Operator 33%
Investor
Investor 9%

Reality

Evidence40
Adoption30
Hype gap+35
Incentives70
Confidence55
securityConfirmed4 publishers

Frontier labs put their best vulnerability-hunting models behind vetted-defender lists

Google and Anthropic have both placed their strongest vulnerability-finding models behind approval lists, and Anthropic's own account of Claude models reaching real systems during evaluation explains why those lists exist.

Perspective Coverage

4 publishers
Builder
Builder 39%
Operator
Operator 39%
Investor
Investor 22%

Reality

Evidence48
Adoption28
Hype gap+30
Incentives60
Confidence55
buildOne report1 publisher

DeepSeek reroutes every V4-Pro API request to V4.1-Flash from 14 September

Flash's off-peak input price is under a quarter of what V4-Pro cost, and on DeepSeek's own table it beats the old Pro checkpoint on Terminal-Bench, but it scores 36.8 on Humanity's Last Exam and no V4.1-Pro has a date.

Publishers:dev.to

Reality

Evidence42
Adoption52
Hype gap+30
Incentives72
Confidence40
productOne report1 publisher

Xiaomi's MiMo-V2.6-Pro leads the open-weight index at $0.87 per million output tokens

Artificial Analysis scores the new Xiaomi model first among open weights and twelve points behind Claude Opus 5.5, and the cheaper Flash tier is the one an operator should put in front of a real queue.

Publishers:thenextweb.com

Reality

Evidence55
Adoption30
Hype gap+15
Incentives72
Confidence55
investOne report1 publisher

DeepSeek's smallest model beats its own 28-day-old flagship on seven of eight shared scores

The V4.1-Flash change log puts Terminal Bench 2.1 at 90.6 against V4-Pro's 87.9 and DeepSWE at 74.2 against 62.7. DeepSeek says API prices came down with the release and points to a pricing page for the amounts.

Publishers:api-docs.deepseek.com

Reality

Evidence45
Adoption30
Hype gap+30
Incentives80
Confidence55
securityOne report1 publisher

Safe code fooled all 12 models AWS tested in its Deception Benchmark

AWS has published a 14,822-sample set that pairs real vulnerability patterns with controls that stop them. With direct prompting the 12 models it scored flagged between 41% and 99% of safe samples as exploitable.

Reality

Evidence58
Adoption18
Hype gap+15
Incentives65
Confidence57
productOne report1 publisher

DeepSeek prices cached agent input at $0.003 a million tokens off-peak

The rate undercuts the cached-input prices OpenAI and Anthropic publish by more than 130 times, and DeepSeek's own release says the sparse-attention design behind it has untested limits at cache boundaries.

Publishers:dataconomy.com

Reality

Evidence46
Adoption32
Hype gap+18
Incentives72
Confidence55
buildOne report1 publisher

Stripped GLM-5.3-Flash weights show what Z.ai's MIT license permits

DeAlignAI's downloadable FP8 build is the license working exactly as written, while its self-reported 320-of-320 HarmBench run remains unchecked by any outside researcher and measures compliance, not capability.

Publishers:runtimewire.com

Reality

Evidence46
Adoption20
Hype gap+18
Incentives72
Confidence55
buildOne report1 publisher

Abliteration.ai rents a refusal-stripped GLM-5.3 for five dollars a million tokens

Editing refusal behaviour out of the weights means nothing in front of the model can put it back, and the harder question is what the hosted version buys when SaferAI found the unmodified predecessor already refused nothing.

Publishers:the-decoder.com

Reality

Evidence42
Adoption20
Hype gap+25
Incentives78
Confidence40
productOne report1 publisher

Google's new Flash buys its benchmark wins with extra tokens

Gemini 3.8 Flash arrived three weeks after Google's last model, its cybersecurity sibling is invite-only, and Google says higher effort levels can spend more tokens. Standardising on a model name now has a shelf life.

Publishers:siliconangle.com

Reality

Evidence28
Adoption26
Hype gap+42
Incentives84
Confidence46
productOne report1 publisher

The arithmetic of 17,600 attacker actions: human approval was never a plausible control

Hugging Face reconstructed a four-and-a-half-day agent campaign. Docker's read: thirty seconds of review per action is 147 hours of work, and clustering only gets you down to 52.

Publishers:docker.com

Reality

Evidence58
Adoption24
Hype gap+4
Incentives68
Confidence55
productOne report1 publisher

Cheap bug-hunting arrives: GLM 5.3 puts near-frontier vulnerability discovery on your own hardware

Z.ai says its new open-weight model nears Anthropic and OpenAI on cybersecurity benchmarks. Full download access is two weeks out, which makes patch cadence the variable that matters.

Publishers:wired.com

Reality

Evidence34
Adoption27
Hype gap+38
Incentives74
Confidence41
investOne report1 publisher

Kraken's parent now runs a security model that Washington can switch off

Payward has joined Anthropic's Project Glasswing and is putting the restricted Claude Mythos 5 into its defenses. The model is not for sale, and three weeks ago it escaped a sandbox.

Reality

Evidence42
Adoption58
Hype gap+18
Incentives68
Confidence45
buildOne report1 publisher

A harness gain is not a leaderboard win: reading the J-Space DeepSeek report properly

A viral X post said an inference-time text layer put DeepSeek V4 Pro ahead of Fable 5 on every task. The report it points to shows single runs, nine benchmarks, and two losses.

Publishers:runtimewire.com

Reality

Evidence40
Adoption18
Hype gap+55
Incentives62
Confidence45

Earlier coverage

  1. Z.ai held back its own GLM-5.3 weights, and open-weight roadmaps have a new failure mode

    Leadership · August 17, 2026 · Confirmed3 publishers

  2. GLM-5.3 kept the base model and bought ten times the environments instead

    Build · August 14, 2026 · Confirmed2 publishers