Four AI firms back one robots.txt signal to block training scrapers

Google, Microsoft, Anthropic and OpenAI have committed to honoring a shared robots.txt directive that lets website owners opt out of having their content scraped to train generative AI models.¹²

What it does

A robots.txt file is a plain text file on a website that tells automated crawlers which pages they may visit. Until now, a publisher who wanted to keep AI training scrapers out had to list every provider’s crawler by name, and a provider running several crawlers needed an entry for each.²¹

The new proposal collapses that into one entry. A site adds a user agent called __usage_ai-training, followed by rules such as “Disallow: /” to block everything, or an “Allow” line to carve out a section, like a press kit.¹ The setting would not apply to models that perform only non-generative tasks.²

The draft was written by Kevin Kelley, Anthropic’s technical lead for standards, and has been submitted to the Internet Engineering Task Force, the standards body that maintains the Robots Exclusion Protocol, the format robots.txt follows.¹²

What it does not do

The signal targets training crawlers only. It does not stop crawling in general, and it does not remove a site from Google search results.¹ For Google, it will be read as a stop sign for the Google-Extended user agent, which gathers material used to train Gemini models.¹ That control is not new: Google introduced Google-Extended in 2023, promising to explore more machine-readable choices for publishers as AI use spread.³

It also cannot undo anything. Pages already crawled stay crawled; the directive only governs future scraping.¹

Microsoft says it will add a separate setting called AI Grounding, for cases where an AI model fetches a page to answer a user’s question rather than to collect training data. It would also reach third-party AI tools built on Microsoft technology. OpenAI plans similar controls.²

Who ships when

Anthropic already supports the directive. Microsoft says it will begin on February 1, 2027. Google says early 2027, and OpenAI says in the coming months.¹² Meta has voiced support for the idea but has not committed to implementing it.²

What publishers should do

Nothing yet, unless you enjoy editing text files ahead of schedule. Once your crawlers of concern announce support, one entry should replace the per-provider list. Keep the existing entries in the meantime, since the old per-provider rules are what work today. Seen it before: a voluntary standard that works only as long as everyone agrees to read it.

The arrangement is also honor-based. Robots.txt has always been a request, not a lock, and the four companies are promising compliance, not building enforcement.

Our read: One shared signal is a real improvement over maintaining a list that grows with every new model. Competitors agreeing on a single line of text is rarer than it should be. The gaps are the usual ones: past scraping stays untouched, Meta is not yet on board, and the dates are staggered into 2027. Watch whether the IETF adopts the draft as written and whether Meta signs on.

Footnotes

1. Engadget, “Google joins OpenAI and Anthropic in promising new scraping controls”. Retrieved October 31, 2026. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. SiliconANGLE, “Google, Anthropic and others to support new AI crawling opt-out standard”. Retrieved October 31, 2026. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

3. Google The Keyword, “An update on web publisher controls”. Retrieved October 31, 2026. ↩