Analysis
Cloudflare announced on July 1, 2026 that beginning September 15, its default settings will block so-called 'mixed-use' AI crawlers — bots that both train models and pull content to answer live user queries — from accessing ad-supported content on new customer websites and existing free-tier customer sites, unless the AI company compensates the publisher. The policy builds on Cloudflare's 2025 Pay Per Crawl marketplace, introducing a new 'Pay Per Use' model that pays publishers specifically when their content demonstrably drives value inside an AI-generated answer, rather than simply charging per page fetch regardless of whether that content was actually used.
Cloudflare framed the move under the banner "Content Independence Day: no AI crawl without compensation," with CEO Matthew Prince stating: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." The company published crawl-to-referral ratios intended to illustrate the imbalance AI companies have created for publishers: Google's crawlers generate roughly 14 page visits for every 1 referral sent back to a publisher's site, a ratio Cloudflare considers a reasonable exchange given Google Search still drives meaningful traffic. OpenAI's ratio, by contrast, is roughly 1,700-to-1, and Anthropic's is roughly 73,000-to-1 — meaning Anthropic's crawlers scrape tens of thousands of pages for every single visit they send back to the original publisher.
Cloudflare also cited data showing more than half of AI crawler traffic re-fetches pages that haven't changed since the last crawl, adding an efficiency argument to the compensation argument: AI companies are not just taking content without proportionate return traffic, they're often re-scraping the same unchanged content repeatedly, generating real infrastructure costs for publishers with no corresponding benefit.
“The shift from an opt-in marketplace to a default-blocking policy is the most consequential part of the announcement.”
The shift from an opt-in marketplace to a default-blocking policy is the most consequential part of the announcement. Under the prior Pay Per Crawl system, publishers had to actively enable payment requirements; under the new default, AI crawlers are blocked from ad-supported content unless payment terms are established, reversing where the negotiating burden sits. Given Cloudflare's scale — routing a significant share of global web traffic — this default change will affect enormous numbers of ad-supported sites simultaneously, regardless of whether individual publishers have the negotiating leverage to strike deals with AI companies on their own.
The competitive and regulatory context includes years of escalating tension between AI companies and publishers, from robots.txt-based opt-outs to Google's narrower 'Google-Extended' crawler-blocking mechanism, alongside ongoing lawsuits from publishers including The New York Times against AI companies over unauthorized training-data use. Cloudflare's policy represents a more systemic, infrastructure-level intervention than any single publisher or lawsuit could achieve independently, given how many sites route through its network.
For publishers and content businesses, Cloudflare's default shift is a meaningful structural change in leverage — for the first time, a major infrastructure provider is making non-payment the exception rather than the norm for large-scale AI content extraction. For AI companies and the founders building on top of their models, this adds a new, potentially significant cost line to any product that depends on live web content retrieval, a cost that previously existed mostly in legal gray area rather than as an enforced default.
What to watch: how AI companies including OpenAI and Anthropic respond ahead of the September 15 deadline, whether they strike broad payment agreements with Cloudflare or route around the policy through alternative crawling infrastructure, and whether other major infrastructure providers (Akamai, Fastly, or hyperscaler CDNs) adopt similar default-blocking, pay-per-use models of their own.
