Publishers Face Three-Tier AI Crawler Economy After Cloudflare Rule Change

The landscape for AI crawlers accessing publisher content has splintered into three distinct tiers, according to new analysis following Cloudflare's latest policy update. One year after the infrastructure provider changed its default settings to help publishers block AI crawlers, the company is now targeting a murkier category: "mixed-use" bots that serve both traditional search and AI training purposes.

Cloudflare announced that starting September 15, its default settings will allow mixed-use crawlers to index pages for search but block them from AI training and agentic tasks on ad-supported pages. The move puts pressure on companies like Google, whose crawler has historically been treated as pure search infrastructure but now feeds multiple AI products.

"We understand how we got here with the mixed-use crawlers, but it doesn't work for the ecosystem," Cloudflare chief strategy officer Stephanie Cohen told Digiday, which first reported the details. "There has to be a better way than forcing people into the decision of all or nothing."

Why it matters

The fragmentation of the AI crawling economy creates asymmetric leverage. Publishers who rely on Cloudflare's rules to protect their content are only addressing the most compliant actors, while well-funded scrapers continue to harvest content through residential proxies and data brokers. This dynamic explains why licensing deals remain concentrated among a handful of major publishers while the broader industry struggles to monetize AI usage of their work.

The gray market problem

Cloudflare's new rules only govern traffic that flows through its network—roughly 20% of the web. Frederick Jahn, whose company Centinel Analytica helps publishers defend against stealth crawlers, estimates that while 20-30% of traffic to major news sites comes from identified crawlers, approximately 25% of total traffic consists of "stealth" crawlers that mimic human users.

"Cloudflare only governs what's behind Cloudflare," noted Chris Dicker, CEO of Candr Media. "Well-funded scrapers can route around it. They can use residential proxies or just buy data from second-hand brokers that have been scraped elsewhere."

Jahn, who previously built the scraping tools he now defends against, argues publishers need maximum-friction defenses: client-side challenges on first page load, systematic blocking of non-essential crawlers, and rigorous vetting of detection vendors. The goal is making scraping expensive enough to force intermediaries into licensing negotiations.

A compliant alternative emerges

One potential counter-model comes from Ceramic, an AI search company founded by former Google engineering VP Anna Patterson. Working with Cloudflare, Ceramic is piloting a premium search API that pays publishers when their content is used to answer AI queries.

The economics hinge on efficiency. Patterson claims Ceramic's costs run around $0.05 per 1,000 queries, compared to $5-$14 for rival search APIs. The system crawls pages once, then serves 100-300 word snippets to language models. Publishers get paid whenever their snippet is sent to a model, regardless of whether the output visibly cites them.

Patterson argues that if enough major publishers adopt metered payment models routed through infrastructure gatekeepers, it could compress the market for third-party scrapers by offering AI companies a cheaper, legally compliant alternative.

Google's dilemma

Cloudflare's September 15 deadline creates particular pressure on Google, whose crawler has served both search indexing and AI training. Cohen declined to promise that the new settings would let publishers block Google's AI training without sacrificing search visibility, noting the company wants to "engage in active dialogue" before the deadline.

Dicker sees implications beyond Cloudflare's network. If the company succeeds in forcing other AI players to split or declare their crawlers, it weakens Google's defense against U.K. regulators who have questioned whether the company should be required to separate its search and AI crawling infrastructure.

The details in this analysis were first reported by Digiday.