AI Watermarking Can't Tell Who Did the Thinking

Major AI companies are embedding invisible watermarks in their outputs as governments worldwide roll out new disclosure requirements. Anthropic watermarks Claude-generated text, Google applies SynthID to Gemini responses, and OpenAI is testing similar techniques for ChatGPT.

The European Union's Artificial Intelligence Act introduced labeling rules this summer requiring companies to mark AI-generated content related to politics, public health, security, and the economy. Content that has undergone genuine human review or editorial control is exempt. China mandates both visible and hidden labels for AI-generated text, images, audio, and video. In the United States, California's AI Transparency Act requires large providers to embed origin disclosures in AI-generated images, video, and audio—though not text—and to offer detection tools.

How watermarking works

AI watermarking exploits patterns in how language models select words. When multiple words fit naturally in a sentence, the system follows a hidden preference—choosing "start" over "begin," for instance. Across longer passages, these micro-choices create a detectable signature that software can recognize.

Google tested SynthID-Text across nearly 20 million Gemini responses and found watermarked answers rated as highly as unwatermarked ones. Anthropic reports the word-choice variations are subtle enough that readers cannot detect them. The technique does not degrade output quality.

Why it matters

Watermarking can only confirm that a large language model was involved—not whether it edited a sentence, checked grammar, or wrote entire sections. This limitation creates serious problems in a globalized workforce where non-native English speakers outnumber native speakers four to one, according to British Council estimates. A Carnegie Mellon study found AI reduced graduate student writing time by 65% and improved quality, with larger gains among second-language English users.

An analyst who spends a week researching, building arguments, and checking evidence might ask AI to polish the prose. The model could select every final word, triggering a watermark—yet the human did all the intellectual work. Employers, publishers, and recruiters who rely on watermarks risk penalizing legitimate use while missing actual cheating.

University of Reading researchers found 94% of fully AI-written exam answers submitted in real assessments escaped detection, demonstrating the genuine challenge institutions face. But watermarks cannot distinguish between a student who outsourced an entire essay and one who wrote it before asking AI to improve the English.

Polish Nobel laureate Olga Tokarczuk faced backlash after acknowledging she asked an AI chatbot for development suggestions, later clarifying she uses AI only for research. The incident illustrates that even as AI becomes commonplace, its use carries reputational cost—regardless of how minimal the assistance.

The fundamental limitation

Watermarking succeeds for certain media types. Fake recordings, cloned voices, or fabricated photographs mislead because people believe the underlying event occurred. Text operates differently. People dictate, translate, prompt, edit, fact-check, and rewrite—often multiple times with multiple tools. Watermarks may simply reveal that a writer needed help with English, not that they cheated.

These details were first reported by Dr. Anda Bologa, Senior Researcher with the Tech Policy Program at the Center for European Policy Analysis, writing in CEPA's Bandwidth journal.