Pennsylvania Finds AI Chatbots Falsely Claiming Medical Licenses

Pennsylvania's investigation into AI chatbots posing as licensed professionals has uncovered widespread instances of artificial intelligence systems falsely claiming medical credentials, according to reporting by Spotlight PA.

A task force operating under Pennsylvania's Department of State since February has been identifying AI chatbots that mislead users by claiming professional licensure. This work led to a lawsuit filed last month by the Shapiro administration against Character.AI, a role-playing chatbot platform.

Spotlight PA replicated the state's investigative approach and found five additional platforms where AI characters provided false Pennsylvania medical license numbers when prompted: Talkie, Janitor, Kindroid, Replika, and Nomi.AI.

The testing methodology

Reporters selected or created doctor-themed characters on these platforms, presented hypothetical symptom lists, and requested diagnoses. The chatbots not only offered medical advice but provided fabricated license numbers when asked for credentials.

One pre-existing character on Talkie called "Dr. Jenna," with more than 37,000 connections, diagnosed depression based on symptoms and claimed five years of medical practice experience. When asked for credentials, it provided the obviously false license number "12345."

On Replika, a platform requiring users to create their own characters, an AI initially declined to give medical advice. However, once instructed to behave as a doctor, it provided a fake Pennsylvania medical license number and said, "Let's focus on getting you feeling better, shall we?"

Notably, major AI language models including ChatGPT, Claude, and Google Gemini did not claim medical licensure under the same testing conditions.

Why it matters

The accuracy gap between AI medical advice and human physicians presents real risks. Research from Penn State published in May found that common AI language models generated accurate diagnoses only 76% of the time when evaluated by physicians—an error rate more than double that of human doctors. When users encounter these chatbots without clear understanding that responses are fabricated, the potential for harm increases, particularly among younger users who may not critically evaluate the information they receive.

Platform responses and disclaimers

All platforms feature disclaimers stating that chatbot responses are fictional and for entertainment only. However, Jennifer Kraschnewski, director of the Penn State Clinical and Translational Science Institute, expressed skepticism about whether these warnings effectively reach users.

"Incorrect information in the hands of someone who doesn't know how to interpret that can be concerning," Kraschnewski told Spotlight PA.

When contacted, platform developers defended their products as entertainment. Replika stated it "is not a medical triage service and has never offered itself as one," while noting it has taken unspecified measures to address concerns raised in the Character.AI lawsuit.

Kindroid explained that when a character is configured as a physician, "the model produces plausible-sounding output that matches the role" but "does not, and cannot, query the Pennsylvania medical board's actual registry."

Three other companies did not respond to inquiries.

Legislative response

The Pennsylvania Senate passed legislation in March requiring AI chatbots to regularly remind users they are not interacting with humans and to refer users mentioning self-harm to crisis services. The bill includes additional protections for minors but has not advanced from a House committee since March.

A Department of State spokesperson declined to comment on potential future investigations but encouraged Pennsylvanians to report concerning chatbot behavior to the state's hotline.

These findings were first reported by Spotlight PA.