Publishers Build Parallel Websites for AI Agents, Not Humans

Major publishers are no longer just blocking AI bots—they're rebuilding portions of their websites specifically for machine consumption, betting that visibility in AI-powered search will matter more than the traffic costs.

Time has converted all its webpages into markdown versions, a stripped-down format that AI systems can process without the HTML, CSS, and JavaScript designed for human browsers. The publisher now blocks all AI bots by default, then whitelists approved bots and redirects them to these simplified pages, according to Mark Howard, Time's chief operating officer.

The move reflects a calculated trade-off: Time is separating bot traffic from human visitors entirely, routing AI agents to content-only feeds while preserving the full page experience—and ad inventory—for people. Howard said the strategy should improve Time's visibility in AI search results and strengthen the publisher's pitch for its GEO product, which helps brands shape their messaging in AI-generated answers.

Competing approaches to agent-readable content

Time is using TollBit, a marketplace connecting publishers and AI companies, to handle the HTML-to-markdown conversion. TollBit claims the process reduces page tokens by 90% on average and cuts fetch time from over a minute to 0.25 seconds. Toshit Panigrahi, TollBit's co-founder and CEO, said removing layout markup helps AI systems comprehend articles more accurately and reduces hallucinations.

The Economist is testing agent-readable versions as well, but only for marketing copy and B2B sales material that already sits outside its paywall, Digiday previously reported. As a subscription publisher, it faces a different calculus: exposing too much content to AI agents could undermine the value proposition for paying subscribers.

A third major news publisher, speaking anonymously, is experimenting with Web Model Context Protocol (WebMCP), a standard co-developed by Google and Microsoft that lets websites share structured data directly with AI agents. The executive said WebMCP could improve citation rates in AI search while reducing content delivery network costs, since serving bot requests through a dedicated layer is cheaper than processing full page loads.

French daily Le Monde is exploring how to surface content to agents while detecting whether those agents represent paying subscribers.

Why it matters

Publishers are making expensive infrastructure bets on a future where AI agents, not search engines, mediate access to information. The shift forces a fundamental question: if an AI bot reads your article but never sends a human to your site, have you gained anything? Time is wagering that high domain authority with AI systems will translate to commercial value, even without the click. But independent consultant Scott Messer argues publishers should only build for agents if they genuinely believe in long-term discovery value—not out of fear of becoming invisible. The answer depends heavily on business model: what makes sense for an ad-funded scale publisher may be ruinous for a hard-paywalled news brand.

The details in this article were first reported by Sara Guaglione for Digiday.