Major AI developers including OpenAI, Google, Microsoft, and Anthropic have announced support for a unified standard allowing publishers to opt out of web scraping across multiple AI crawlers simultaneously. Proposed by Anthropic technical lead Kevin Kelley within the Internet Engineering Task Force (IETF), the proposed “policy token” resides in a site’s robots.txt file to streamline preferences that previously had to be set individually per company.

While Anthropic has already implemented the signal, Google and Microsoft plan to adopt it next year, and OpenAI expects to implement it in the coming months. Meta’s communications head Andy Stone also expressed support for the common signal, though the company has not confirmed if or when it will honor it.

The unified token addresses growing publisher frustration with AI web scraping. Microsoft noted its granular controls will let sites opt out of AI training or grounding without losing Bing search visibility, mirroring similar safeguards previously implemented by Google.

Why it matters

  • Streamlines AI opt-out management for web publishers into a single robots.txt standard rather than managing separate crawler rules.

  • Ensures content creators can block AI scraping without sacrificing search engine indexing and traffic from Bing or Google.

  • Demonstrates industry convergence on web standards as scrapers face increasing legal pressure and blocking from platforms like Cloudflare.

Source: theverge.com