Four of the largest AI companies say they will honor a single opt-out signal from publishers. OpenAI, Google, Microsoft and Anthropic have agreed to adopt a shared standard that lives in a site’s robots.txt file. The proposal was first reported by Search Engine Roundtable.
The mechanism is called a “policy token.” It was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force. Today a publisher must configure AI training preferences separately for each company’s crawler. Under the new approach, one declaration covers all of them.
Why publishers wanted this
Websites have spent two years chasing individual crawlers. A site owner had to know that GPTBot belongs to OpenAI, Google-Extended belongs to Google, and ClaudeBot belongs to Anthropic. New crawlers appeared every few months, and each one needed a separate rule.
That is an operational burden for small publishers in particular. A newsroom with no engineering staff cannot track every user agent. A shared token turns a moving target into one line of configuration.
The catch: honoring it is voluntary
Nothing in the announcement makes this binding. The companies say they will restrict their crawlers when they encounter the token, but compliance is self-declared. Meta’s communications head Andy Stone signaled support and called a common signal “an important step.” He did not say whether Meta will honor it, though.
That gap is why other approaches keep appearing. The Really Simple Licensing standard lets publishers demand payment from crawlers. Cloudflare now blocks AI crawlers by default for new domains. Both are enforcement mechanisms rather than polite requests.
What it means for traffic
The bigger fight is not scraping but referral traffic. Reddit’s chief executive said this week that Google’s AI Mode does not drive traffic the way search results once did. Reddit’s stock fell sharply afterward. Sites that block AI crawlers may protect their content and lose the last trickle of visitors at the same time.
A single opt-out token makes the choice clearer, and clearer choices cut both ways. Publishers who want to be read by AI systems can now say so just as easily as publishers who want out.
Source: The Verge. Related reading: Etsy’s new rules on AI listings.
The Netherlands, for internationals
One email a week: the week's news for internationals in the Netherlands, new practical guides and what changed in the cities. No spam, unsubscribe any time.
