▸ MarTech/AdTech · Cloudflare
Cloudflare sets September 15 default block on AI training crawlers
Cloudflare rolled out new controls that classify AI crawlers into search, agent, and training use cases, available to every customer including the free tier. Starting September 15, training and agent crawlers will be blocked by default on pages that display ads, and mixed-use crawlers like Googlebot get caught by training blocks unless they separate their bots.
The bottom line: Audit how much of your content discovery now depends on AI crawlers, and confirm your own site's crawler settings before the September defaults flip.