Analysis
Cloudflare CEO Matthew Prince said bot traffic has overtaken human traffic online faster than he expected, with Cloudflare Radar showing bots now account for 57.4% of HTTP requests globally versus 42.6% from humans, according to Fortune. Prince's conclusion for where the web goes from here: "Clearly it's going to be pay to crawl."
A Forecast That Arrived Early
Prince had previously projected bot traffic would exceed human traffic by 2027; Cloudflare's own data shows it crossed that threshold in May 2026, well ahead of schedule. Much of the acceleration comes from AI crawlers ingesting web content to train models -- traffic that, as Prince frames it, consumes a site's bandwidth and compute without sending a reader, or any revenue, back to the publisher. Pulse has previously covered Cloudflare's earlier pay-per-crawl policy push; this week's disclosure is the clearest numeric evidence yet that the underlying trend it responded to has only accelerated.
The Infrastructure Play Behind The Framing
Cloudflare's "Disallow AI Training" setting lets a website remain indexed for search while blocking AI crawlers from using its content for model training -- a middle path between total openness and blocking search entirely. Apple, Google and Microsoft have all agreed to respect the setting, a meaningful concession from three of the largest AI-model operators to a policy set by an infrastructure provider rather than negotiated bilaterally with individual publishers. Cloudflare has gone further, unveiling a digital wallet designed to let AI agents pay a fraction of a penny per request -- infrastructure meant to make "pay to crawl" an actual functioning payment rail rather than just a licensing negotiation.
Why This Matters Beyond Cloudflare
Cloudflare sits in front of a large share of global web traffic, which gives Prince's numbers unusual authority relative to any single publisher's anecdotal complaints about AI scraping. If bots really do reach 1,000 times human traffic within five years, as Prince projects at the current growth rate, the basic economics of running a content business -- built for decades on the assumption that a human reader eventually sees an ad or converts -- shift meaningfully toward a model where AI-agent traffic itself has to become a direct revenue source, not just an operating cost to absorb.
Cloudflare's Own Incentive
The bear case: Cloudflare has an obvious commercial incentive to frame the problem in a way that positions its own new payment infrastructure as the solution, and "pay to crawl" as a widely adopted standard requires AI labs broadly agreeing to pay for access they can currently take for free in many cases -- something Apple, Google and Microsoft doing so for search-indexing exceptions doesn't yet prove will extend to full training-data licensing at scale. Whether smaller AI labs and startups, without Big Tech's negotiating leverage or legal budgets, adopt the same payment rails or simply route around Cloudflare-protected sites is the open question that determines whether this becomes an actual internet-wide norm or stays a policy Cloudflare enforces only where it has direct infrastructure control.