Meta's commercial terms, effective October 30, appear to let it scrape sites tied to business accounts for AI training, according to Nieman Lab. With no opt-out, a publisher negotiating payment for its archive is bargaining with a counterparty that may already hold a contractual route to the same text.
Perspective Coverage
5 publishers
- Builder
- Builder 17%
- Operator
- Operator 51%
- Investor
- Investor 32%
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+30
- Incentives50
- Confidence62
buildOne report1 publisher Meta's Commercial Terms take effect October 30 and ask Facebook business users to permit crawling of their websites for AI training. Meta says robots.txt controls stay as they are, but the permission sits in a contract.
Reality
- Evidence55
- Adoption
- Not scored
- Hype gap+10
- Incentives
- Not scored
- Confidence50
buildOne report1 publisher One author's October 12 crawl of 740 top Tranco sites found that 69 of the 109 blocking GPTBot also block ChatGPT's search or browsing agent. Those sites opted out of ChatGPT answers as well as training, and the crawl cannot say how many meant to.
Reality
- Evidence55
- Adoption15
- Hype gap0
- Incentives30
- Confidence60
buildOne report1 publisher Cloudflare blocks AI agent and training traffic by default on ad-carrying pages of domains added since September 15, 2026. Search crawlers stay allowed, so a browsing assistant can be refused a page that a search indexer reads freely.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+10
- Incentives
- Not scored
- Confidence40
buildOne report1 publisher Pub-trivia.app's developer warns that Disallow: /dashboard/ would block every dashboard page except the bare /dashboard its homepage footer links to. Blocking a fetch never stops indexing, so the noindex tag has to go live before the Disallow.
Reality
- Evidence45
- Adoption
- Not scored
- Hype gap+5
- Incentives
- Not scored
- Confidence55
Anthropic told an Australian inquiry a broad copyright exemption is off the table and proposed opt-out 'conditional approval' for AI training instead. The ABC and SBS want the onus on AI firms, so the inquiry is now deciding who pays to police training bots.
Reality
- Evidence58
- Adoption
- Not scored
- Hype gap+15
- Incentives75
- Confidence60
buildOne report1 publisher FoxyInvoice prerenders every public Angular route to static HTML at build time so crawlers that skip JavaScript get the full page text. A build step and a one-line drift check take the place of the always-on SSR server its team declined to run.
Reality
- Evidence40
- Adoption15
- Hype gap+20
- Incentives70
- Confidence40
buildOne report1 publisher Fewer than 1% of Cloudflare sites block search crawlers while 17% block training. A single mixed-use crawler used to collapse those two into one decision, and the new setting splits it for three named operators.
Reality
- Evidence55
- Adoption35
- Hype gap+20
- Incentives70
- Confidence60
An ecommerce audit consultant writing in Entrepreneur says most AI crawlers read only a page's initial HTML, so a product page that ranks in Google can return nothing to an assistant fielding around 50 million shopping queries a day.
Reality
- Evidence24
- Adoption33
- Hype gap+38
- Incentives72
- Confidence45
Amazon's robots.txt now disallows ChatGPT-User and OAI-SearchBot, and in ZDNET's tests the shopping agent sent a buyer to Walmart, eBay and a retailer called Soaplicity instead of Amazon listings.
Reality
- Evidence64
- Adoption66
- Hype gap+24
- Incentives72
- Confidence58
New disallow rules for ChatGPT-User and OAI-SearchBot went into Amazon's robots.txt without an announcement. Amazon earns roughly $56 billion a year from advertising that bills only while shoppers are on its own pages.
Reality
- Evidence68
- Adoption58
- Hype gap+22
- Incentives74
- Confidence66
buildOne report1 publisher Google-Extended and Applebot-Extended are robots.txt tokens that never issue a request, so the nine agents OpenAI, Anthropic and Perplexity document are the only AI traffic a WordPress site can confirm for itself.
Reality
- Evidence74
- Adoption48
- Hype gap−8
- Incentives68
- Confidence60
buildOne report1 publisher Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.
Reality
- Evidence62
- Adoption40
- Hype gap−8
- Incentives68
- Confidence55
buildOne report1 publisher Cloudflare's Disallow AI Training control lets Googlebot and Applebot keep indexing a site while a directive Cloudflare writes into robots.txt tells them not to train on it. Bingbot is expected to honor it in early 2027.
Reality
- Evidence30
- Adoption25
- Hype gap+25
- Incentives70
- Confidence35
buildOne report1 publisher Eight hand-kept lists sat behind every new page on pub-trivia.app. One typed array now feeds the sitemap, breadcrumbs and footer, with unit tests that fail the commit when a description runs long.
Reality
- Evidence58
- Adoption14
- Hype gap+12
- Incentives40
- Confidence55
buildOne report1 publisher The AI Act's general-purpose AI obligations became enforceable in August, and the parts of them that touch a data pipeline depend on records only the collector can write while it is fetching. Penalties scale with turnover.
Reality
- Evidence32
- Adoption
- Not scored
- Hype gap+22
- Incentives70
- Confidence44
buildOne report1 publisher With no completed sales, nine live asks running from £35 to £240 and no route to the £26.99 shop price, a self-hosted pricing tool proposed £35.39 and recorded the missing check only in a log line.
Reality
- Evidence50
- Adoption12
- Hype gap+6
- Incentives30
- Confidence55
buildOne report1 publisher A dev.to post sets out two ways assistant retrieval fails while ordinary search keeps working, one in robots.txt rules written per user-agent and one in copy that appears only after JavaScript runs. It offers no measurement.
Reality
- Evidence30
- Adoption
- Not scored
- Hype gap+15
- Incentives
- Not scored
- Confidence40
buildOne report1 publisher The file exists so an agent can skip your documentation and read the install steps instead. Pandex checked 8,565 of them, found 237 references pointing at names anyone could register, and registered one.
Reality
- Evidence38
- Adoption52
- Hype gap+18
- Incentives62
- Confidence44
buildOne report1 publisher One developer's cache-header diff read perfectly and still shipped a dead year-long cache, which is the argument for granting agents autonomy only where git revert plus a redeploy is the whole undo path.
Reality
- Evidence52
- Adoption12
- Hype gap+8
- Incentives22
- Confidence50
Earlier coverage
- hreflang fails as a set, not a page, and your per-page audit will never see it
Build · August 25, 2026 · One report1 publisher
- Cloudflare's Bot Preference Sync makes robots.txt an output of the edge, not a policy of its own
Build · August 21, 2026 · Confirmed2 publishers
- Cloudflare's one-click AI block names GPTBot, not the bot that decides if ChatGPT cites you
Build · August 18, 2026 · One report1 publisher
- The AI-training bans live on the big infrastructure blogs, not the small publications
Build · August 17, 2026 · One report1 publisher
- Three files, three contracts: robots.txt, sitemap.xml and llms.txt are not rivals
Build · August 15, 2026 · One report1 publisher