Reuters and Time Block AI Bots by Default, Whitelist Approved Crawlers

Publishers reverse their bot-blocking strategy

Reuters and Time have implemented a fundamental shift in how they manage AI web crawlers, moving from blocking specific bots to blocking all bots by default and maintaining whitelists of approved crawlers. Both publishers made the change last month, joining People Inc. and The Atlantic in adopting this more restrictive approach.

The strategy reflects growing publisher frustration with the imbalance between the value AI companies extract from news content and what they provide in return. "Our content costs money to create. It has significant value, and the access to it, we feel, must be earned," said Josh London, head of Reuters Professional.

Time currently allows approximately 70 bots to access its content, according to COO Mark Howard. These range from crawlers operated by major AI labs and social platforms to automated systems Time uses for its own website operations. The publisher uses ScalePost to manage AI bot access.

Why it matters

This whitelist approach creates negotiating leverage for publishers at a critical moment. As AI companies scale their training operations and launch products that directly compete with news sites for traffic, publishers need mechanisms to force licensing conversations. The friction strategy—even if imperfect—increases scraping costs and makes unauthorized access more difficult, potentially driving AI companies to the negotiating table rather than simply taking content.

The friction strategy

Reuters implements its blocks using robots.txt files, a method that relies on voluntary compliance. A Tollbit report found that 30% of AI bot scrapes in Q4 2025 ignored explicit robots.txt permissions. But perfect enforcement isn't the goal.

"Friction is really the goal," said Lindsay Van Kirk, People Inc.'s SVP of innovation, speaking at an IAB Tech Lab event on May 28. When People Inc. switched from a blocklist to an allowlist, the company went from blocking roughly 2,100 user agents to over 30,000, illustrating the scale of bot traffic publishers face.

Adding latency and forcing scrapers to use proxy networks cuts into their profit margins. "Every scraper who has to pay a home proxy network in order to get access to the content is margin that you are taking out of their business," Van Kirk explained.

Approved bots must offer value

Reuters allows bots that provide what London calls a "fair value exchange" across four categories: licensing, traffic, site function, and monetization. Bots that drive subscription or advertising revenue get approved, as do those necessary for Reuters' ad tech infrastructure. The publisher's robots.txt file shows it permits some crawlers from Amazon, Google, Bing, Yahoo, and ChatGPT.

The approach hasn't hurt site traffic so far, London noted. After two years of monitoring bot activity, Reuters had sufficient data to identify which bots could be blocked without impacting revenue.

Cost savings and ongoing challenges

Blocking hundreds of bots also delivers cost benefits. "The cost of the bot-blocking vendor can be almost covered by the decrease in non-human traffic that you experience," said Alphonse Hardel, head of agency at Reuters. The reduction in serving content to unauthorized bots created measurable savings.

The strategy requires constant maintenance as bots evolve. "It's not a set it and forget it approach," London said. Van Kirk noted that People Inc. briefly broke all email syndications when implementing the change because automated link-checking systems were inadvertently blocked.

The challenge extends beyond known AI companies. A growing economy of third-party scrapers operates beneath major AI labs, taking content without disclosure for resale. "Someone is stealing from you and reselling it," Van Kirk said. "How do you think about that as opposed to whether or not you want to show up in ChatGPT? Those are two fundamentally different questions."

Reuters has secured AI licensing agreements with companies including Microsoft and Meta. The publisher combines its bot-blocking approach with monitoring tools, server-level enforcement, commercial licensing arrangements, and legal protections.

The IAB Tech Lab published a bot-blocking guide last month to help website owners implement similar strategies. The SPUR Coalition, a publisher group formed this year with major news organizations, announced it added 30 members last week to develop technical standards for AI licensing and content protection.

These details were first reported by Digiday.