Amodei gives his botnet warning a 6-12 month deadline
A sharptext.net column argues the loudest AI risk estimates should be scored against the record of the people issuing them. Of the four forecasts it assembles, only one can be checked inside a year.
The Board Room · Leadership desk
What happened
- A sharptext.net column reports that an Alignment Science lead at Anthropic puts the chance his company's technology leads to human extinction in the next decade at greater than 10%.
- It sets that estimate beside Anthropic chief executive Dario Amodei's own stated 10-25% chance that AI will have a catastrophic impact.
- The column recalls that in 2019 Amodei and other OpenAI researchers believed GPT-2 was too dangerous to release to the world.
- On the Hugging Face incident, it says the behaviors were not emergent, the reporting from safety organisations was arguably too dramatic, and the fallout was exceptionally limited.
- Its author argues the frontier AI community should be treated as fringe activists who are sometimes right but often wrong.
Why it matters
- Who has to decide decision A forecast with a 6-12 month window can be entered in next year's review and marked right or wrong. A probability spread over a decade never comes due inside a planning horizon. The two cannot be treated as the same input to a budget.
- What it limits constraint Both probability figures a board is likely to be handed come from one company's payroll, so citing them together does not give it two independent estimates of the same risk.
- What doesn't add up contradiction The column doubts the messengers while conceding the capability: it expects recursively self-improving AI soon and disputes only volition and recourse. Those doubts about the messengers do not support deferring technical controls.
- What it costs cost Managers absorb the work of answering staff who have heard the extinction figure without its time frame, and what this column gives them to answer with is an argument about credibility.
Next story · Invest · Oct 9
OpenAI and Anthropic executives war-game the backlash from an AI cyberattack insiders expect within a yearConfirmed4 publishers
Anthropic's chief executive put a clock on his own warning. Writing last Saturday, Dario Amodei said he worries that in 6-12 months an agent swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage, and that the scale of damage would keep increasing from there if AI becomes more powerful without the necessary guardrails [1].
Only the botnet forecast can be scored without waiting a decade. Of the four Amodei forecasts the column assembles, three carry dates: 2019 on GPT-2, last year on employment, last Saturday on the botnet, alongside an undated 10-25% chance of catastrophic impact [13]. The extinction estimate the column attributes to an Anthropic Alignment Science lead runs out to the next decade [2]. A board approving security spend this quarter can put the botnet forecast back on the agenda next autumn. The other three sit outside any cycle a board runs.
The column's argument about whom to trust has two halves, and they are not equally usable. One half is record and incentive: the author writes that today's AI leaders "have their own spectrum of social and financial incentives, and most importantly, they are not scientific authorities worthy of deference by default" [7]. The other half is character. "some people are just crazy. And crazy people are not worth indulging if you'd like to stay sane," he wrote [8]. Anyone can check a track record against the dates. Judging temperament is another matter, and an employee who has just read a double-digit extinction probability gets no answer from being told that the person who said it is unbalanced.
On specific scenarios the column argues physical feasibility. Building a bioweapon is far harder in practice than the scenario assumes [15]. Nuclear launch chains use multi-factor authentication before launch, almost all of it involving multiple humans, and even Russia's dead hand program reportedly has to be activated by a person [18]. Most large companies and financial institutions use air-gapped backups against exactly the records-destruction case, the author writes, adding that not all of that work is public [19]. Two of those three are answerable inside a single firm: whether the backups are air-gapped, and who can reach them.
On capability the column agrees. Its author writes that he is sure some version of recursively self-improving AI will exist soon if it does not already, and puts his disagreement on whether such agents would have volition or malicious designs of their own, and whether humans would have recourse if they did [17].
Both probability figures come from one payroll [14]. An Anthropic alignment lead supplies the extinction number and Anthropic's chief executive supplies the 10-25% range, so a reader who cites the pair has cited one employer twice. The author describes himself as a relative tech outsider who nevertheless pays close attention to this world [10]. A Pod Save America host wrote this week that he takes very seriously the warnings of the people who are actually building the technology, and the column says he has it precisely backwards [9].
What to watch
- Whether the persistent-botnet forecast holds by the end of the 6-12 month window Amodei set last Saturday.
- Whether the Anthropic alignment lead's greater-than-10% extinction estimate is ever published with a stated method behind it.
- Whether safety organisations' reporting on the next containment incident matches the limited fallout the column describes at Hugging Face.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption30
- Hype gap+35
- Incentives65
- Confidence50
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Anthropic CEO Dario Amodei wrote last Saturday: "it's my worry that in 6-12 months [an agent] swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails."
- [2]
The column reports that an Alignment Science lead from Anthropic thinks there is a greater than 10% chance that his company's technology will lead to human extinction in the next decade.
- [3]
In 2019, Amodei and other OpenAI researchers believed that OpenAI's GPT-2 model was too dangerous to release to the world.
- [4]
Amodei warned last year that AI could "soon" eliminate half of entry-level jobs and lead to 20% unemployment.
- [5]
Amodei himself has said that there is a 10-25% chance that AI will have a catastrophic impact.
- [6]
The column argues: "We ought to normalize treating the frontier AI community as fringe activists who are sometimes right but often wrong."
- [7]
The column writes of today's AI leaders that they "have their own spectrum of social and financial incentives, and most importantly, they are not scientific authorities worthy of deference by default."
- [8]
The column's author wrote: "some people are just crazy. And crazy people are not worth indulging if you'd like to stay sane."
- [9]
A Pod Save America host wrote this week that he takes very seriously the warnings of the people who are actually building this technology, and the column's author says he has it precisely backwards.
- [10]
The column's author describes himself as a relative tech outsider who nevertheless pays close attention to this world.
- [11]
The column says millions of normal people, too busy with jobs and families to track the pattern, are understandably freaked out when they hear that agents have gone rogue and formed a conspiracy, or hear the Anthropic extinction estimate.
- [12]
The column's author says he was castigated by a Sharp Tech emailer who was upset that he had not spoken up on behalf of people terrified by AI risk.
- [13]
Of the four Amodei forecasts the column cites, three are dated (2019 on GPT-2, last year on employment, last Saturday on the botnet swarm) and one, the 10-25% chance of catastrophic impact, is given without a time window.
- [14]
Both probability figures the column reports come from Anthropic personnel: the extinction estimate from an Anthropic Alignment Science lead and the 10-25% catastrophic-impact range from Anthropic's chief executive.
- [15]
The column says that a rogue AI creating bioweapons that kill billions would be very difficult in practice, and that it is worth reading up on how difficult.
- [16]
On the Hugging Face incident, the column says the behaviors were not emergent, the reporting from safety orgs was arguably way too dramatic, and the fallout was exceptionally limited.
- [17]
The column's author writes that he is sure some version of recursively self-improving AI will exist soon if it does not already, but is not at all clear that such agents will ever have volition or malicious designs of their own, or that humans will have no recourse if they do.
- [18]
The column argues that state-backed nuclear arsenals are designed with multi-factor authentication processes prior to launch, almost all of which involve multiple humans, and that even Russia's dead hand program reportedly has to be activated by a human.
- [19]
The column says most large companies and financial institutions use air-gapped backups to prevent catastrophic fallout from an attack on digital records, and that not all of this work is public.
Sources
2 independent publishers whose own reporting we read for this story.
- Some of All Fears
sharptext.net
1 article · September 18, 2026
- vox.comThe AI threat keeping scientists up at night
1 article · September 19, 2026
Topics and entities
Topics
- AI Agents
- Forecast Accountability
- Biosecurity
- AI Existential Risk Debate
- AI Safety Policy