Cloudflare's New Default Policy Forces AI Crawlers to Pay Publishers for Content
Cloudflare announced on July 1 that starting September 15, its default settings will block 'mixed-use' AI crawlers from scraping ad-supported content on new customer sites and existing free-tier sites unless the AI company pays, building on its 2025 Pay Per Crawl marketplace with a new 'Pay Per Use' model that compensates publishers when their content actually drives value in an AI answer. CEO Matthew Prince cited crawl-to-referral ratios as extreme as 73,000-to-1 for Anthropic and 1,700-to-1 for OpenAI, versus 14-to-1 for Google.
Jul 1, 2026