What did OpenAI, Google, Microsoft and Anthropic agree to?
On 30 October 2026, OpenAI, Google, Microsoft and Anthropic each said they will honor the same machine-readable signal in robots.txt. The token is __usage_ai-training. A website can use it once to state an AI training preference to several providers. That replaces a separate rule for every company.
The announcements landed on the same day, and each company published its own explanation. The Verge reported the change. A new Internet-Draft submitted to the Internet Engineering Task Force (IETF) defines the token and its syntax.
Google framed the change as relief for site owners. "This will help websites express their AI training preferences across multiple AI providers in a single step," the company wrote. It published the line in its Search Central blog. The post is dated 30 October 2026 and signed by John Mueller, a Google Search Advocate.
The practical effect is narrow but real. A publisher that wants to refuse training can now say so in one place. Each provider still decides when to read that preference and how to apply it.
Each company described a different scope. Google tied the signal to its Google-Extended control and to Gemini Apps. Microsoft tied it to training and to grounding in Bing, Copilot and Microsoft 365. OpenAI tied it to foundational generative AI training. The draft, written by an Anthropic standards lead, supplies the syntax that all four say they will read.
The change affects anyone who runs a site with a robots.txt file. That group includes news outlets, blogs, retailers, documentation sites and small publishers without legal teams. It also affects AI developers, who must update their parsers, and SEO specialists, who advise clients on crawl rules.
A basic opt-out uses the ordinary robots.txt grammar. The draft's example gives this group:
- •User-Agent: __usage_ai-training — names the shared purpose instead of a single crawler.
- •Disallow: / — expresses a preference against training for the whole site.
- •Allow: /press-kit/ — keeps an exception for a path the site wants to promote.
How does the __usage_ai-training token work?
In the draft, that group expresses a preference against AI training for every path except /press-kit/. The file still uses the parser rules crawlers have followed for decades. There are no new directive names and no new grammar.
The token behaves like a product token. Consumers match it exactly, apart from case. The double underscore prefix reduces the chance of an accidental match with a real crawler name.
Allow and Disallow carry the meaning. A matching Disallow means the site has expressed a preference against training for that path. A matching Allow means the site has expressed a preference in favor. If the file has no group for the token, then no preference has been expressed at all.
The path rules allow fine control. A site can refuse training on news archives while allowing it on marketing pages, or the reverse. Because the draft reuses longest-match precedence, the most specific path wins, just as it does for crawler rules today.
Wildcard groups are excluded. A group written for User-Agent: * must not be read as matching the policy token. Many sites use a wildcard group to block everything. Under this draft, that group says nothing about AI training.
The signal is separate from crawl control. "This policy token is not a directive for crawling behavior," the draft states. A provider may use the preference to inform crawling, but the document does not require it to stop fetching a page.
Precedence follows specificity. A rule aimed at a particular purpose should beat a broader one. Private agreements can override the file, which leaves room for paid licences and custom deals.
The IETF is the standards body that published the Robots Exclusion Protocol as RFC 9309 in September 2022. Its AI Preferences working group is building a shared vocabulary for how sites express choices about AI use. The draft for the new token tracks that vocabulary instead of inventing a separate one.
The working definition is narrow. AI training means using an asset to modify the learned parameters of a generative AI model. Training for models used only in non-generative tasks is excluded, even if those models can also generate content.
The group is not finished. Its vocabulary draft, draft-ietf-aipref-vocab-08, is dated 13 September 2026. A companion draft covers how to attach usage preferences to content in HTTP, which suggests robots.txt will be one channel among several.
The process matters for readers. An individual draft becomes a standard only after working group adoption, review and consensus. The token draft is at the first step. Its acknowledgments list contributors from Google, Microsoft, OpenAI, Cloudflare, the BBC, the Financial Times and News Corp. The list also includes the Internet Archive and the Electronic Frontier Foundation. Publishers and model builders negotiated the format together, but agreement on this slice does not settle the whole vocabulary.
Because robots.txt is already an IETF standard, the new token can ride on existing parsers. A crawler that does not implement the token will treat its group as an unknown user agent and ignore it. That backward compatibility is the main technical reason the group chose this design.
When will each AI company honor the opt-out?
The four companies are not moving at the same speed. A site owner who publishes the token today will see different behavior from each provider. The effective date matters as much as the announcement.
Microsoft is adding more than a shared token. It will let sites override the robots.txt training preference at page level. It is also separating training from grounding, the use of content to inform generated answers. Its grounding control has three settings: Allow, Disallow, and a new DisallowExtended. The new option limits use to Microsoft-operated experiences such as Copilot and Microsoft 365. It excludes third-party services Microsoft supplies.
A transition period covers sites that still rely on NoArchive and NoCache. A temporary grounding preference entered in Bing Webmaster Tools applies until the page is recrawled. The preference also ends on 30 September 2027 at the latest. After 1 March 2027, the old tags will no longer be read as training or grounding preferences.
Google's implementation has a condition. Google applies a default when a site uses the shared token. If the file does not mention Google-Extended, Google plans to treat the site as if Google-Extended were disallowed. Google says sites that specifically allow Google-Extended remain eligible for grounding in Gemini Apps. Google-Extended has existed since 2023 and does not affect inclusion in Search or act as a ranking signal.
OpenAI says the new signal will not replace its existing crawler controls. Those controls will continue to apply and be respected. The company also urged other providers to honor the signal, a hint that adoption is not yet universal.
- •Anthropic: already adopted the signal, according to The Verge. The draft's author, Kevin Kelley, is Anthropic's technical lead for standards.
- •Microsoft: honors the new training and grounding preferences from 1 February 2027. NoArchive and NoCache stop being read after 1 March 2027.
- •Google: intends to support the signal in early 2027, according to its crawler documentation. Its blog says the company aims to support the control in 2027.
- •OpenAI: aims to implement the signal in the coming months. The company says it will publish implementation details later.
Why does a shared opt-out matter for publishers?
The old system was fragmented by design. Each provider named its own crawler, and each publisher had to track those names and add a rule for every one. A shared token reduces that maintenance work to a single group.
The pressure behind the change is easy to see. Publishers have watched AI crawlers collect their pages for training and for generated answers. Some have responded with technology rather than text. Cloudflare now blocks AI crawlers by default for new customers. The Really Simple Licensing (RSL) standard gives publishers a way to set terms and charge for access.
A preference signal sits between those options. It is softer than a block and cheaper than a lawsuit. It also fits a regulatory trend. Google says the proposal is consistent with the EU Code of Practice for general-purpose AI under the AI Act. That code treats robots.txt as a primary way for publishers to express machine-readable preferences.
Small publishers gain the most from the simpler format. A newsroom without a legal team can add one group and reach several providers. Large publishers already negotiate private deals, and they may still prefer contracts to a public flag.
Microsoft's announcement draws the line it wants publishers to see. "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work," the company wrote. It also said the preferences do not establish licensing or payment arrangements.
That last sentence matters for anyone hoping the token will produce revenue. It records a choice about use. It does not price the content, enforce a contract or create a debt.
Training and grounding draw on the same crawlers but serve different purposes. Training changes model weights over months. Grounding retrieves pages at answer time, so a page can influence a response without being in the training set. A site that cares about one use but not the other needs separate rules. That is why Microsoft built a separate grounding preference.
Blocking and licensing are not mutually exclusive. A site can block one crawler, license content to another and publish a shared training preference for the rest.
What are the limitations of the new signal?
The first limitation is enforceability. The signal is voluntary on both sides. A provider can sign up, publish a date and still change its crawler behavior later. The draft says a consumer may use the preference to inform crawling. It does not say must.
The second is status. The draft is not an IETF standard, and the working group has not finished its vocabulary. The document expires in May 2027 unless it is renewed. Companies have adopted a moving target, which is workable but not stable.
The third is scope. The shared token covers training only. It says nothing about grounding, search indexing or the retrieval that feeds answers. Microsoft's separate grounding controls show the gap. A site that opts out of training may still appear inside generated answers.
The fourth is a set of traps in the syntax. A wildcard group does not count, so a site-wide block may leave training preferences unstated. Google gives an explicit Google-Extended rule priority over the shared token. Microsoft lets a page-level setting override the site file. A stale rule can quietly cancel a new choice.
The fifth is timing. The signal cannot undo training that already happened. It applies to future collection under each provider's schedule. Publishers who want to affect models already in development have to rely on contracts, law or blocking.
The sixth is verification. There is no independent auditor for the token. A publisher can check logs for crawler visits, but not whether a model was trained on a page. Cloudflare's default block is a blunter instrument, but it produces a visible access denial rather than a promise.
None of this makes the preference retroactive or portable across providers. Each provider implements the lookup separately, so bugs and delays are likely in the first months.
What should site owners do now?
Site owners can prepare now without waiting for every provider. The first step is to decide what the site wants. Training, grounding and search indexing are separate questions, and the new controls treat them separately.
Testing is manual for now. Site owners can inspect crawler logs for visits from each provider. They can also check whether pages still appear in generated answers. No dashboard reports whether the token was read.
- •Audit the current robots.txt file and list every AI-related group and tag.
- •Publish an explicit __usage_ai-training group; do not rely on a wildcard.
- •Keep a separate Google-Extended rule if the site wants to stay eligible for Gemini Apps grounding.
- •Use page-level overrides only where a section needs different terms.
- •Record the dates when each provider says its support begins.
What happens next?
The calendar is the next thing to watch. Microsoft's training and grounding preferences take effect on 1 February 2027, and its old tags stop counting after 1 March 2027. Google's documentation points to early 2027. OpenAI has promised months, not a date.
The IETF process is the longer watch. The draft expires on 3 May 2027, and the working group still has to settle the wider vocabulary and any objections. Adoption by four large providers gives the format momentum, but it does not make it a standard.
Watch for other providers as well. Meta, Amazon, Mistral and xAI did not join this announcement. If they stay outside, the token will still require per-provider rules for a large part of the market. Regulators may also treat the signal as evidence of publisher choice under the EU AI Act. That would raise the cost of ignoring it.
The story is a coordination fix, not a settlement. It makes an opt-out easier to express. It does not answer the harder question of whether publishers should be paid when their work improves a model. That argument will continue in courts, contracts and standards meetings long after the robots.txt line is added.
For readers, the near-term effect is modest. Answers may cite fewer sources as sites opt out, and publishers may gain a clearer record of what they asked for. The value of that record depends on how providers behave after the dates they have published.