AI Giants Back Standard Letting Publishers Block Web Crawlers

WHO:

Engadget - Google, OpenAI, Anthropic, and Microsoft pledged to support a common web-crawler signal.

Techmeme / The Verge - Emma Roth reported that OpenAI, Google, Microsoft, and Anthropic said they would adopt the standard.

Inc - The article describes the companies as “The 4 AI Giants.”

WHAT:

Engadget - The companies pledged to support a new signal for `robots.txt` that blocks scraping for AI training.

Techmeme / The Verge - The standard would let publishers opt out of AI scraping across multiple crawlers.

Techmeme / The Verge - The new control would let publishers apply their AI training preferences across multiple crawlers.

Inc - The companies agreed to honor a single `robots.txt` signal.

Inc - The article says the agreement does not solve the broader issue of scraping data from the internet that has already occurred.

HOW:

Engadget - The proposed control would work through a common signal in `robots.txt`.

Techmeme / The Verge - Publishers could use one standard to express preferences across multiple AI crawlers.

WHY:

Techmeme / The Verge - The standard is intended to let publishers set AI-training preferences across multiple crawlers.

Inc - The article notes that honoring the signal does not address data already scraped from the internet.

(UPDATED October 31, 2026 16:02 UTC)

Google, OpenAI, Anthropic, and Microsoft pledged to support a shared web-crawler signal. Publishers could use one robots.txt standard to set AI-training preferences across multiple crawlers. The control can block future scraping, but won’t address data already scraped from the internet.