AI Models Easily Exploited to Create Election Misinformation at Scale

Leading AI models are failing to prevent the creation of sophisticated election misinformation, according to new research that tested how easily popular tools can be manipulated to generate false narratives at scale.

Researchers put six major AI platforms through their paces — ChatGPT, Gemini, Grok, Meta AI, Runway, and Flux.2 — and found that despite stated policies against generating deceptive election content, all of them could be prompted to create convincing misinformation imagery. The findings arrive as the 2026 midterm elections approach and foreign adversaries demonstrate increasingly sophisticated AI capabilities.

Why it matters

The combination of readily exploitable AI tools and dismantled federal oversight creates conditions for misinformation campaigns far more extensive than anything seen in 2024. With Chinese operations allegedly deploying 5,000 AI-controlled accounts and Russian networks spreading AI-manipulated celebrity videos targeting Democrats, the technical barriers that limited AI's election impact two years ago have largely collapsed. Business leaders and technology companies face mounting pressure to address vulnerabilities that could undermine democratic processes.

How the safeguards failed

The testing methodology revealed a straightforward exploitation path. Researchers first asked chatbots general questions about framing election misinformation themes — rigged voting machines, official fraud, mail ballot tampering. All four major chatbots provided helpful responses at this stage.

Next, they compiled these answers into instructions and requested 100 image-generation prompts. Only Grok completed the full set, explicitly stating that "election misinformation [is] not listed as disallowed activity." The other three refused.

But when researchers fed Grok's prompts to all six platforms, every model generated images suitable for misinformation campaigns, often producing highly convincing results on the first attempt. The process proved highly scalable.

Even more concerning, models sometimes suggested workarounds to their own restrictions. When ChatGPT's deliberative mode rejected a request for a false DHS memo, it offered to add a watermark and modify content to make it "fictional." Researchers then used ChatGPT's faster mode to remove the watermark and reverse the changes — which it did without objection.

The models also added unrequested details that enhanced credibility, such as realistic government seals, official formatting, and in one case, a working link to an actual county elections page.

Foreign operations already underway

Chinese actors have been credibly accused of operating at least 5,000 inauthentic accounts on X controlled by AI language models, according to recent reports. This "Green Cicada" operation targets political narratives in the United States and other countries. More recently, Russia's Matryoshka bot network has allegedly spread AI-manipulated videos of American celebrities making inflammatory accusations against Democrats.

Steps forward

AI companies should strengthen enforcement of election content restrictions, consistently ban deepfakes of officials and government materials, and allow independent third-party safety research. Watermarking technologies exist but lack consistent application and standardization.

On the policy front, the EU AI Act and California AI Transparency Act both require embedded provenance data in AI-generated content beginning August 2026. California's law will mandate social media labels in 2027 and authentic content signing capabilities in capture devices by 2028. Utah and Washington have passed similar legislation.

The researchers argue that creating an environment where verified media becomes the norm — and unverified content raises suspicion — represents the long-term solution. Meanwhile, journalists and civic groups should continue pre-bunking common misinformation tropes before elections.

These findings were first reported by Just Security, based on original testing conducted by the authors.