Skip to content

Benchmark

AA-Briefcase

Artificial Analysis private long-horizon knowledge-work benchmark reporting turn counts and input-token consumption per task.

Known aliases

  • AA Briefcase
  • AA Briefcase v1.1

Current stories

investConfirmed3 publishers

xAI holds Grok's $2 token price for a model 40 Elo points behind Fable 5.1

Grok 4.7 arrived on Monday after five walked-back timelines, with 40% more parameters and Grok 4.6's list price intact. Cursor's own cost chart still puts its price per task above GPT-6 Astra and Claude Sonnet 5.

Perspective Coverage

3 publishers
Builder
Builder 42%
Operator
Operator 27%
Investor
Investor 31%

Reality

Evidence45
Adoption35
Hype gap+15
Incentives70
Confidence55
investOne report1 publisher

Grok 4.7 buys 5.9 points of CursorBench at Grok 4.6's token price

xAI built Grok 4.7 on a larger base model with a longer reinforcement-learning run and kept the API at Grok 4.6's rates. The open question for buyers is how many tokens the longer runs burn.

Reality

Evidence34
Adoption22
Hype gap+28
Incentives82
Confidence46
scienceOne report1 publisher

Token efficiency absorbs GPT-6 Astra's 2.5x price increase inside the coding harness

Artificial Analysis priced the new model at $10 and $50 per million tokens, and the threefold token cut it measured in the Codex agent harness covers that increase where the roughly 10% cut on its intelligence suite does not.

Reality

Evidence63
Adoption
Not scored
Hype gap+16
Incentives62
Confidence57
scienceOne report1 publisher

A 91-task benchmark for agent knowledge work runs every task cold, and says so

Artificial Analysis' AA-Briefcase grades deliverables built from four multi-week projects, but each task starts with no memory of the model's own earlier submissions. Continuity stays unmeasured.

Reality

Evidence60
Adoption
Not scored
Hype gap+12
Incentives62
Confidence55
scienceOne report1 publisher

Frontier scores at $2/$6: Grok 4.6 ties GPT-5.6 on one evaluator's index for a fifth the output price

Artificial Analysis puts Grok 4.6 at 61 on its Intelligence Index, level with GPT-5.6 Sol, at $2/$6 per million tokens. The same pages record 48 seconds to first token.

Reality

Evidence56
Adoption24
Hype gap+24
Incentives63
Confidence50
buildOne report1 publisher

Four frontier models in four days, and the cheapest number in your agent plan has an expiry date

Grok 4.6, Gemini 3.7 Flash, DeepSeek V4 Pro and GLM-5.3 all chase agents that stay on task. The pricing underneath them is moving faster than the benchmarks.

Publishers:dev.to

Reality

Evidence58
Adoption55
Hype gap+12
Incentives68
Confidence48
buildOne report1 publisher

Artificial Analysis moves eval onto your data, and turns model choice into procurement

Optima lets buyers build benchmarks from their own datasets and agent traces, then scores candidate models on quality, cost per task and time per task.

Publishers:the-decoder.com

Reality

Evidence34
Adoption16
Hype gap+22
Incentives71
Confidence33
buildConfirmed4 publishers

Three frontier launches in a day, all pitched on price. Open weights set the ceiling.

Grok 4.6, Qwen3.8-Max and DeepSeek V4-Pro shipped inside about 24 hours, and two of the three came with downloadable weights. The benchmarks existed to justify a cheaper invoice.

Perspective Coverage

4 publishers
Builder
Builder 41%
Operator
Operator 31%
Investor
Investor 28%

Reality

Evidence68
Adoption52
Hype gap+22
Incentives74
Confidence63