Cloudflare Demands AI Firms to Classify Crawlers or Face Bans

WHO:

Techmeme - Cloudflare is the organization setting the deadline for AI companies.

TechCrunch - AI companies are the entities required to comply with Cloudflare's new policy.

WHAT:

NBC News - Cloudflare is implementing a policy that requires AI crawlers to be categorized by September 15, or they will face blocking on publisher sites.

Techmeme - Cloudflare demands AI companies to differentiate their web crawlers into categories: search, AI training, and AI agents.

TechCrunch - AI companies risk being blocked by default on many publisher sites if they do not comply with the new policy.

WHEN:

Techmeme - The deadline set by Cloudflare for AI companies to comply is September 15.

WHY:

TechCrunch - The new policy aims to push AI companies to pay for publishers’ content.

HOW:

Techmeme - Cloudflare's policy requires a clear separation of web crawlers used for different purposes.

(UPDATED July 2, 2026 23:25 UTC)

Cloudflare is rolling out a new policy that mandates AI crawlers to be categorized by September 15, or they will be blocked from accessing publisher sites. This move aims to enhance the management of web traffic and ensure compliance with publisher guidelines. The deadline is fast approaching, prompting many in the tech industry to prepare for these changes.

--

(UPDATED July 2, 2026 11:52 UTC)

Cloudflare has announced a new policy that requires AI companies to categorize their web crawlers into distinct groups, including search, AI training, and AI agents. If these companies fail to comply by the deadline of September 15, they risk being blocked by default on numerous publisher sites, pushing them to pay for access to content. This move aims to ensure that AI companies are held accountable for the content they utilize from publishers.