Ninth State of AI Report credits better tooling with a 13-point coding gain on an older model

A budget can act on the size of that gain. Changing the tools, context and feedback around GLM-5.1 added 13 percentage points of success on 100 SWE-bench Verified tasks ^[8]. In relative terms, the model solved about 25% more tasks ^[9]. Its weights were not changed ^[3].

A skeptic would call this thin evidence: one controlled study, 100 tasks, on a model the report admits is dated. The author saw the objection coming and wrote that, although this is an older model, the principle holds ^[16]. We think the objection is right about the size of the gain and wrong about its direction. We would not expect 13 points to carry over intact to other models or to work outside coding ^[8]. The direction has better support, because the study kept the model fixed and changed only what surrounded it ^[3].

The author applies the same argument to people. "My view is that much of the gap between people getting substantial value from AI and those getting little comes down to knowing how to use it and how to set it up," the author wrote ^[5]. The introduction calls this a "human skill issue," not an "AI technology issue," and proposes "a Genius Bar for AI, where people can get hands-on help choosing tools and setting them up to help them in their everyday tasks" ^[5][6]. The author has invested in companies building and applying AI since 2013 ^[4]. That stake does not make the diagnosis wrong. The introduction presents it as a view, and its main published support is the GLM-5.1 study ^[5][3].

Public scores give a buyer less to go on when picking a model. In maths, scientific reasoning and coding, benchmarks meant to challenge models for years are reaching their ceilings within months, according to the report ^[12]. Capability can also go backwards. The author found Opus 5 "a step back" from Opus 4.6, and Opus 5.5 good again ^[13]. A multi-year exclusive signed this quarter because of one leaderboard position relies on a ranking the report labels temporary ^[2].

Both adoption figures in the introduction come from AI labs. Anthropic's internal index measures its own model's share of its own R&D work, and the finding on non-developers comes from a study by OpenAI ^[15][14]. The author is more cautious about where automated research goes from here. Karpathy's autoresearch automates a useful part of research, the author wrote, but "sustained, fully autonomous recursive improvement remains to be demonstrated" ^[7].

The report is in its ninth annual edition ^[1]. It calls itself "the most widely read and trusted analysis of key developments in AI" ^[10] and grades its own forecasts ^[11]. The introduction lists three past calls as examples and says DeepSeek-R1 proved one of them right by surpassing OpenAI's o1 ^[11]. The introduction does not include the full scorecard, so the material does not show the report's hit rate.

In our view the evidence justifies paying for setup work at least long enough to measure its effect on a team's own tasks. Suppose that test finds gains close to the GLM-5.1 study's 13 points ^[8]. Next quarter's upgrade decision then changes. A new model would have to beat the current model after tuning, and its price would be judged against that margin.