Topic
Task-level evaluations used to compare open and proprietary models.
Six coding agents pass 73.6% to 91.3% of LSCBench tasks, but the best writes working, secure code on only 25.1%, NUS and Alibaba researchers found. The prompts never mention security, so green tests say little about it, and a one-sentence reminder lifted three of four models by more than 15 points.
Reality
Gemini 4 Argon beats Claude Opus 5.5 on 14 of 18 benchmarks in Google DeepMind's published scores, according to a dev.to comparison. Opus takes the other four, including Terminal Agents at 66.4 to 57.4.
Reality
Liquid AI says its LFM2.5-24B-A2B scores 48.09 on LiveClaw against 38.19 for the larger Qwen3.6-35B-A3B. The 184 tok/s it reports on an Apple M5 Max is a decode rate, so agent latency depends on how many tokens the model spends thinking.
Perspective Coverage
4 publishersReality
ATLAS, a 26-model long-context benchmark, has Gemini 3.1 Pro Preview leading at 128K tokens and Claude Opus 4.6 leading at 1M. A shortlist drawn at 128K cannot be assumed to hold for prompts near 1M.
Reality
Anthropic launched Claude 4.5 Opus on October 9 with sub-agent spawning in the model and a 500,000-token context, per a dev.to write-up. The claimed SWE-bench Verified lead over GPT-6.1 Sol and Gemini 2.5 Ultra comes from that post alone.
Reality
Anthropic's Claude Sonnet 5.5 resolves 70.6% of Terminal-Bench 4.0 tasks against 66.4% for Opus 5.5, a dev.to post reports. If that gap holds on a team's own tasks, the mid-tier model is the one to test first as the default for terminal agents.
Reality
Alibaba's new 27B open-weight model defaults to its deepest reasoning setting. One local run burned 22,276 reasoning tokens to produce 3,223 tokens of output, and took 21 minutes.
Perspective Coverage
6 publishersReality
DeepSeek V4.1 Flash costs $2.10 off-peak for an agent run that costs $75 on Claude Opus 5, about 36x less, according to a dev.to analysis. Paying peak-hour rates halves that gap to about 18x, and running at max effort cuts it to about 15x.
Reality
Meta, Duke and UC Davis researchers lifted Gemini 3 Flash from 46% to 62% on Olympiad math by searching for a better harness around the same model. That puts the wrapper on an operator's budget ahead of a bigger model, at least for math.
Reality
Nine LLMs completed a Kaggle benchmark of 120 URLs, 45 of which Python's urlsplit and fetch() resolve to different hosts. If a model approves the Python reading and the request goes out through fetch(), the API key reaches a host nobody approved.
Reality
Ridge Security's 96-test pen-test benchmark had Claude Opus 4.6 reach 63% coverage at $217 a run, against 52% for Gemini 3 Flash at about $5.42. Ridge argues tooling matters more than the model, yet its published runs held the tooling fixed and varied the model.
Reality
Ten LLMs' Blender 5.0 scripts ran only 70% of the time when a Kaggle benchmark executed them in 5.0, against 91% for scripts targeting 3.6. Renamed and removed APIs look like valid code, so the benchmark grades each answer in the exact build the prompt named.
Reality
Nine AI models gave 1,782 commit-credit answers in a test that changed only the user's incentive, and some models moved their answers under that pressure. It matters wherever the assistant that wrote the code also writes the footer that credits it.
Reality
Google's third Flash release in six weeks keeps the $0.75/$3.75 rate card. But the model also spends more tokens per task. Both numbers in your cost model are moving before the price even changes.
Perspective Coverage
8 publishersReality
Astra for Law is GPT-6 Astra with a legal search index and analysis instructions wrapped around it. OpenAI will also sell that index through an API to Harvey and Legora, at a price it has not published.
Reality
Flash's off-peak input price is under a quarter of what V4-Pro cost, and on DeepSeek's own table it beats the old Pro checkpoint on Terminal-Bench, but it scores 36.8 on Humanity's Last Exam and no V4.1-Pro has a date.
Reality
xAI built Grok 4.7 on a larger base model with a longer reinforcement-learning run and kept the API at Grok 4.6's rates. The open question for buyers is how many tokens the longer runs burn.
Reality
Sol lists at $2 and $10 per million tokens and Luna at $0.10 and $0.50. The per-task savings OpenAI published come mostly from the lower price, and the cheaper tier scores below its predecessor on computer use.
Reality
Microsoft and Omdia put the average security operations centre at close to 3,000 alerts a day with 42% never investigated, while a 2026 benchmark scored the best frontier model at 3.8% on finding malicious events in raw Windows logs.
Reality
Bonsai 2 27B keeps 98 percent of Qwen3.8 27B's aggregate benchmark score, up from 95 percent for PrismML's first release, and TechCrunch reports the 5.9GB file is small enough for a PC and possibly a high-end phone.
Reality
Product · September 15, 2026 · One report1 publisher
Leadership · September 3, 2026 · One report1 publisher