UN Demands AI Child Safety Pledge as Abuse Material Surges 260-Fold

UN Pushes for Binding AI Child Safety Standards

UN Secretary-General António Guterres opened the first Global Dialogue on Artificial Intelligence Governance in Geneva this week with a direct call for an AI Child Safety Pledge, declaring that "no child should be a guinea pig for unregulated AI." The initiative marks a shift from abstract AI governance debates toward concrete rules that few governments can afford to oppose.

Guterres outlined three specific requirements for AI systems accessible to minors: child-specific safety testing with external oversight before deployment, technical measures to prevent generation of child sexual abuse material (CSAM) with mandatory detection and reporting, and immediate connection to human support when a child expresses crisis or self-harm intent rather than continuing synthetic conversation. He emphasized that human accountability must remain final—companies cannot deflect responsibility by blaming algorithms.

The urgency stems from measurable harm. The Internet Watch Foundation reported a more than 260-fold increase in AI-generated child sexual abuse videos in 2025, warning that simplified tools now enable offenders to create realistic abuse material without specialized technical knowledge.

Why it matters

Child safety represents the first AI use case where global regulatory consensus appears achievable. While governments remain divided on copyright, compute access, and competition policy, protecting minors from manipulation, sexualization, and crisis mishandling creates rare bipartisan and international alignment. This consensus could establish the template for how AI systems are tested, audited, and held accountable across other domains.

Regulatory Action Accelerates Across Jurisdictions

Europe's AI Act, now in force, already prohibits manipulation and exploitation of vulnerabilities. A 2026 update proposes explicit bans on AI systems generating non-consensual sexually explicit content and CSAM, including nudification applications.

In the United States, child safety has broken typical partisan gridlock. The House passed the Kids Internet and Digital Safety Act on June 29, 2026, requiring platforms to provide tools limiting addictive features and maintain policies against sexual exploitation. The Senate had previously passed the Kids Online Safety Act 91-3 in 2024. Senators Adam Schiff and John Curtis introduced the SAFE KIDS Act in June 2026, targeting chatbot risks specifically—requiring safeguards before bots reach minors, restricting targeted advertising to children, and banning sexual deepfakes.

Companies Implement Safeguards Ahead of Mandates

AI developers are moving preemptively. OpenAI introduced parental controls for ChatGPT in September 2025, allowing account linking and stronger protections including limits on graphic content and romantic role-play for teen users. Character.AI took more dramatic action in October 2025, removing open-ended AI chat for users under 18 and building separate creative experiences based on stories and videos rather than conversational interaction.

Meta faced scrutiny after Reuters reported in August 2025 that internal policy documents permitted chatbots to engage children in romantic or sensual conversations. The company subsequently added training to prevent its AI systems from discussing romance, self-harm, or suicide with underage users.

Implementation Challenges Remain

The practical application presents technical and policy tensions. Age verification protects children but raises privacy concerns. Parental monitoring helps families but may prevent teens from seeking confidential support. Content filters can block abuse material yet also suppress legitimate health information when poorly designed. Rapid takedown requirements risk overreach into protected speech.

Effective child-focused AI regulation requires technical safety testing protocols, appeal mechanisms, crisis escalation procedures, failure documentation, and independent review—more akin to product safety standards than content moderation. Companies must demonstrate how their systems respond when a young user expresses suicidal ideation or when a model attempts to prolong inappropriate conversation.

The Geneva dialogue does not produce binding treaties but aims to develop initial guidelines through a UN-backed expert process. The political momentum, however, is clear: child safety is forcing AI governance from theoretical frameworks into enforceable law.

These details were first reported by Ron Schmelzer at Forbes.