OpenAI, Google, Microsoft and Anthropic will honor one robots.txt line that opts your site out of AI training

A new IETF draft written at Anthropic defines a shared robots.txt token, __usage_ai-training, that tells participating AI companies not to train on your pages. Anthropic already honors it, Microsoft starts February 1, 2027, Google plans to in 2027 and OpenAI in the coming months. It covers training only.

If you publish a company blog, docs or a newsletter archive, one block in your robots.txt file will soon tell OpenAI, Google, Microsoft and Anthropic not to train AI models on it. Today you need a separate rule for each company’s crawler. The four companies said on October 30 they will honor a shared signal instead.

What changed

The signal is a “policy token” called __usage_ai-training, defined in an IETF Internet-Draft by Kevin Kelley of Anthropic, dated October 30. You write it as if it were a crawler name. The draft’s own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That says no AI training on the whole site except the press kit folder. The draft adds no new commands, so older robots.txt parsers simply ignore the block. “AI training” here means using your pages to change a generative model’s weights. It does not cover search indexing or a chatbot fetching your page to answer a question.

Who honors it, and when:

Google says that when a site opts out with the shared token and doesn’t name Google-Extended, it will treat that as a Google-Extended block. Existing company rules keep working, so you can still allow one company and block the rest. Microsoft is going further: from February 1, 2027 it adds page-level training and “grounding” preferences (grounding is when Copilot or Bing’s AI answers pull from your page), and both replace its old NoArchive and NoCache tags. Microsoft says none of these choices affect whether you appear in regular Bing results.

What it means for your site

Decide training and AI answers separately. The shared token only stops training. Blocking it doesn’t remove you from AI answers, and that’s the part that can send you clicks. Google says sites that allow Google-Extended stay eligible to appear and get clicks through grounding in the Gemini app. Microsoft warns that blocking grounding means “reduced brand visibility in AI experiences.”

For most founders, the cheap move is to add the block now if you don’t want your writing in training data. Anthropic reads it today, and the others will pick it up without you editing the file again. Keep any existing rules for GPTBot, Google-Extended and the rest until all four have shipped support.

Treat it as a preference with no enforcement behind it. The draft is an individual proposal that the IETF working group still has to finish, and it expires May 3, 2027 unless renewed. It only binds companies that choose to honor it. Crawlers outside this group can still ignore robots.txt, which is why Cloudflare blocks AI crawlers by default and why publishers are trying paid licensing with the RSL standard, as The Verge notes.