OpenAI released 719 manuscripts of claimed solutions to hard math problems this week and included the model's chain of thought for only ten of them. The advisory group of mathematicians OpenAI consulted had set verification standards the lab's release misses on several counts.
Perspective Coverage
3 publishers
- Builder
- Builder 44%
- Operator
- Operator 38%
- Investor
- Investor 18%
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+40
- Incentives55
- Confidence60
OpenAI released hundreds of results on open problems in math and theoretical computer science, many already verified in the Lean programming language. Those are all but certain to be correct, so the months of reading ahead are about whether they hold new ideas.
Perspective Coverage
5 publishers
- Builder
- Builder 33%
- Operator
- Operator 55%
- Investor
- Investor 12%
Reality
- Evidence62
- Adoption
- Not scored
- Hype gap+35
- Incentives70
- Confidence60
OpenAI published 722 math manuscripts across 372 result families, all produced by an internal model it has not released. About 300 come with Lean formalizations a machine can check, Crypto Briefing reported, so the rest wait on human reviewers whom one mathematicians' group is urging to stop working with OpenAI.
Perspective Coverage
3 publishers
- Builder
- Builder 30%
- Operator
- Operator 47%
- Investor
- Investor 23%
Reality
- Evidence55
- Adoption20
- Hype gap+35
- Incentives65
- Confidence55
OpenAI posts 719 math papers from an unreleased model a week after its advisers asked labs to stop
OpenAI posted 719 math papers from an unreleased model on October 6, a week after its IAS-based advisers asked labs to stop such testing. Three mathematicians showed the same day that its machine-checked Navier-Stokes proof proves less than the paper claims, so its results now rest on ordinary peer review.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+30
- Incentives60
- Confidence40
buildOne report1 publisher Mathematicians' group chaired by Terence Tao calls for an OpenAI boycott over 700 AI proof files
Mathematicians in a group chaired by Terence Tao are urging a boycott of OpenAI after it released more than 700 AI-generated proof files at once. At issue is whether a problem counts as solved before a human has checked and understood its proof, and who does that work.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+30
- Incentives65
- Confidence40
buildConfirmed6 publishers OpenAI posted 372 results from an internal model to GitHub, at about three hours of ChatGPT Pro compute each on average. Finding proofs is now cheap enough that the slow step is mathematicians checking and judging them.
Perspective Coverage
6 publishers
- Builder
- Builder 44%
- Operator
- Operator 31%
- Investor
- Investor 25%
Reality
- Evidence58
- Adoption10
- Hype gap+35
- Incentives72
- Confidence60
OpenAI posted 722 model-written math manuscripts in 372 families on GitHub, many with Lean proofs a computer can check. A mathematician citing one has to learn first whether it is a checked proof or a paper still waiting for review.
Perspective Coverage
10 publishers
- Builder
- Builder 40%
- Operator
- Operator 27%
- Investor
- Investor 33%
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap+45
- Incentives70
- Confidence60
buildOne report1 publisher OpenAI publishes ten math results from its internal Astra model with Lean proofs attached
OpenAI published ten math results from an internal Astra variant, with Lean-formalized proofs and the model's narration of each solution. Astra has no announced API, so what outside teams can take is how the work was split between the model, human authors and a proof checker.
Reality
- Evidence40
- Adoption5
- Hype gap+5
- Incentives55
- Confidence40
OpenAI's louder rollout made its independent math advisers look like company appointees
OpenAI's louder announcement of a nine-member independent math advisory group caused widespread confusion over whether the company had appointed it. Its first job is helping release scores more results from an unreleased OpenAI model, a wave researchers already dread.
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap+20
- Incentives60
- Confidence50
buildConfirmed2 publishers The company says an internal model cleared more than 100 open problems after a month of training, and the advisory group it is backing at the Institute for Advanced Study has been handed the release schedule to work on.
Reality
- Evidence30
- Adoption
- Not scored
- Hype gap+55
- Incentives70
- Confidence55
Every task OpenAI lists for its new mathematics panel is advice
The nine members get early access to OpenAI's research, work unpaid and are free to publish what they tell the company. The researchers who spoke to The Verge still cannot say what happens when they disagree with it.
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+28
- Incentives72
- Confidence55