Four passing runs out of 80 separate first from second on Specific's private-code benchmark

Specific's published leaderboard reports that Fable 5.1 running through Claude Code led Real-SWE with a 38.8% resolution rate.

GPT-6 Astra with Codex CLI followed at 33.8%.

Gemini 3.8 Flash with Gemini CLI scored 31.2% and GLM 5.3 using Claude Code placed fourth at 28.8%.

Specific uses each model's native coding harness, measuring a model-and-harness combination instead of holding the agent scaffold constant; a score can capture differences in tool use, context handling and orchestration alongside model capability, and the leaderboard should be read as a comparison of complete coding setups.

For eight repository-backed sample tasks, Specific reports a median instruction length of 1,742 characters and a median reference solution touching 11 files.

Short prompts leave substantial discovery work to the agent.

Specific defines resolution rate as pass@1 averaged across eight independent runs per task.

Specific's published analysis shows ten tasks, eight model-and-harness combinations and eight attempts for each pairing, producing 640 rollouts.

Specific's published analysis says six of the ten analyzed tasks had aggregate resolution rates below 15%.

The supplied release does not disclose the full task set or enough methodology for outside researchers to reproduce the result.

Public coding benchmarks can expose repositories, issues and eventual patches that may enter model training data, while a private codebase forces an agent to read the system in front of it and infer conventions from the available workspace.

runtimewire.com writes that public coding scores can reward familiarity with visible repositories, and that Real-SWE tests unfamiliar production systems while giving Specific a route from evaluation failures to training-data customers.

Y Combinator's company page says Specific acquires and licenses operational data and codebases.

YC backed Specific in its Fall 2025 batch and lists the company as founded in 2025 with a two-person team.

Specific says its sample codebases include an events product with more than 200,000 users, a consumer fintech platform that has processed more than 100,000 bank statements, and enterprise sales software; the businesses remain anonymous, so those descriptions and usage figures are Specific's claims.

runtimewire.com writes that the low resolution rates point to a practical risk for businesses deploying coding agents, because a plausible patch can still apply the wrong tax rule, break an existing workflow or update code the running service never calls.

One sample assignment asks an agent to repair invoice taxation across businesses with different tax configurations, accounting for customer exemptions, buyer destinations, European VAT registrations and separate sandbox and production tax services, and to record rates and totals on invoices and report settled sales to an outside authority.

A single task moving from failed on all eight runs to passed on all eight moves a combination's score by 10 percentage points.

The two entries using Codex CLI, GPT-6 Astra and GPT-5.6 Sol, finished 17.6 points apart.

Each task is attempted 64 times across the eight combinations, so an aggregate rate below 15% means nine passes or fewer.

Ten tasks at eight runs each give each model-and-harness combination 80 rollouts, so one rollout is worth 1.25 percentage points and the leading 38.8% corresponds to 31 passes out of 80.

GPT-6 Astra's 33.8% is 27 passes out of 80, four passing runs behind the leader's 31.

The spread from first place at 38.8% to eighth at 16.2% is 22.5 points, or 18 rollouts out of 80.

The median sample task supplies about 158 characters of instruction per file the reference solution changes.