Skip to content

Topic

Vendor-reported benchmarks

Self-published evaluation tables and the difficulty of verifying frontier-parity comparisons.

Current stories

buildOne report1 publisher

WEKA's NeuralMesh reaches federal buyers on Tradewinds with its 20x claim still unchecked

WEKA Federal's NeuralMesh won Awardable status on the Department of War's Tradewinds marketplace, so federal buyers can skip one technical review. Its claimed 20x training speedup comes with no published baseline, so agencies will have to test it on their own workloads.

Publishers:runtimewire.com

Reality

Evidence35
Adoption8
Hype gap+55
Incentives70
Confidence55
buildOne report1 publisher

HyperPod's inference gateway scores KV cache and LoRA residency before it picks a pod

Amazon's new EKS managed addon reads Prometheus metrics from every model pod and sends each request to the one with room in its KV cache. The advertised 82% cut in first-token latency rests on a single 4.4-second baseline.

Publishers:aws.amazon.com

Reality

Evidence46
Adoption14
Hype gap+38
Incentives88
Confidence52
productOne report1 publisher

TypeSafe leaves stealth with a model whose confidence scores developers have to calibrate themselves

TypeSafe's first model, Jev, answers structured questions with typed output and a confidence measure attached. The account of its launch says developers have to check those scores against real outcomes on their own data first.

Publishers:siliconangle.com

Reality

Evidence26
Adoption7
Hype gap+48
Incentives80
Confidence57
buildOne report1 publisher

Lambda ran 19 Blackwell nodes on the power budget provisioned for 16

The one measured number behind NVIDIA's tokens-per-megawatt pitch comes from Lambda's Blackwell cluster, where power reclaimed from static provisioning ran three extra nodes. Amazon's Annapurna Labs and d-Matrix got a line each.

Reality

Evidence33
Adoption45
Hype gap+44
Incentives90
Confidence60
buildOne report1 publisher

Cognichip's one detailed ACI example divides out to 40 to 50 times faster

Faraj Aalaei's Cognichip argues that semiconductor physics belongs inside the model, with agents left to orchestrate. The one speedup it has documented in detail is a 55-page spec handled in days against a baseline Cognichip supplied itself.

Publishers:runtimewire.com

Reality

Evidence33
Adoption18
Hype gap+42
Incentives74
Confidence57
investOne report1 publisher

Compute scarcity meters the model OpenAI says can fill out forms at superhuman speed

Astra scored 98.6% on ARC-AGI-3 where its predecessor managed 7.8%. OpenAI is still rationing access while it scales capacity, and that tells a clerical-automation budget more than the benchmark does.

Publishers:fortune.com

Reality

Evidence36
Adoption21
Hype gap+41
Incentives79
Confidence47
buildOne report1 publisher

Polars 2.0 routes every lazy collect() through the streaming engine by default

Ritchie Vink is selling 2.0 as a boring upgrade. Mostly it is, but engine="auto" now resolves to streaming by default, so joins and group_bys stop carrying incidental row order, and pipelines that leaned on it will drift quietly.

Publishers:runtimewire.com

Reality

Evidence62
Adoption20
Hype gap+10
Incentives78
Confidence58
productOne report1 publisher

Google's new Flash buys its benchmark wins with extra tokens

Gemini 3.8 Flash arrived three weeks after Google's last model, its cybersecurity sibling is invite-only, and Google says higher effort levels can spend more tokens. Standardising on a model name now has a shelf life.

Publishers:siliconangle.com

Reality

Evidence28
Adoption26
Hype gap+42
Incentives84
Confidence46
productOne report1 publisher

Dyson hands the flossing decision to a camera inside the toothbrush

The CameraJet spots a gap between two teeth and fires rinse into it inside 100 milliseconds, which leaves the only proof that the machine learning worked sitting inside a phone app the toothbrush does not need.

Publishers:wired.com

Reality

Evidence24
Adoption9
Hype gap+58
Incentives78
Confidence56
buildOne report1 publisher

Masked diffusion sampling spends whole-sequence passes to escape the token-by-token loop

Mercury reports over 1,000 tokens per second per user, and Nemotron Diffusion claims 2 to 8 times autoregressive throughput. Both are vendor figures. The sampler that produces them charges for its speed in arithmetic.

Publishers:dev.to

Reality

Evidence38
Adoption42
Hype gap+30
Incentives62
Confidence40
buildOne report1 publisher

Nvidia's power pitch: 40,000 Rubin GPUs in 100MW, or 2.5kW a GPU at the meter

The zettaFLOPS figure is an estimate. The measured one, sitting in a GB300 example, is a 25 percent density gain, and it is paid for out of the safety margin.

Publishers:tomshardware.com

Reality

Evidence42
Adoption15
Hype gap+34
Incentives78
Confidence55
buildOne report1 publisher

Nvidia's 88-core Vera bets agentic serving runs out of bandwidth before it runs out of cores

One SKU, one compute die, 1.2 TB/s of LPDDR5X, and benchmark wins over a 96-core EPYC. The comparison with AMD's Venice is being set on Nvidia's terms rather than on core counts.

Publishers:tomshardware.com

Reality

Evidence38
Adoption12
Hype gap+33
Incentives78
Confidence44
buildOne report1 publisher

A 27B Apache-2.0 model in 17GB makes local inference a wiring decision, not a demo

Qwen 3.8 27B ran on a MacBook Pro from a 17GB GGUF and spent 21 minutes on one SVG. Licensing and access stopped being the blocker; latency and KV cache budgeting became the job.

Publishers:dev.to

Reality

Evidence48
Adoption58
Hype gap+18
Incentives62
Confidence52