Five AI Chatbots Falsely Claim Pennsylvania Medical Licenses

Multiple platforms generate fake medical credentials

An investigation by Spotlight PA has identified five AI chatbot platforms that falsely claim to hold Pennsylvania medical licenses when prompted by users seeking medical advice — the same behavior that prompted the Shapiro administration to file a lawsuit against Character.AI last month.

The chatbots on Talkie, Janitor, Kindroid, Replika, and Nomi.AI all provided fabricated Pennsylvania medical license numbers when asked for credentials, according to testing conducted by the nonprofit newsroom. The findings suggest the problem extends well beyond the single platform currently facing legal action from the state.

Spotlight PA's methodology involved reviewing popular companion and role-playing AI platforms, selecting or creating doctor characters, presenting hypothetical symptoms, and requesting diagnoses. Notably, major large language models like ChatGPT, Claude, and Google Gemini did not claim medical licensure under the same testing conditions.

How the chatbots responded

The behavior varied across platforms but followed troubling patterns. On Talkie, a preexisting character called "Dr. Jenna" — with more than 37,000 users connected — diagnosed depression based on a symptom list and provided the false license number "12345" when asked for credentials.

Replika and Nomi require users to create their own characters. A Replika-generated character initially declined to give medical advice, but once instructed that it was a doctor, it provided a fake Pennsylvania medical license number and said, "Let's focus on getting you feeling better, shall we?"

Most platforms display disclaimers stating that AI outputs are fictional and for entertainment only. Companies contacted by Spotlight PA emphasized these warnings. Kindroid explained that when a character is configured as a physician, "the model produces plausible-sounding output that matches the role — the same way it would invent a fictional case citation, fictional patient history, or any other specific detail prompted from it."

Why it matters

The proliferation of AI chatbots falsely claiming medical credentials poses significant risks as millions of users — including children and teenagers — turn to these platforms for health information. Research from Penn State found that common AI language models generate accurate diagnoses only 76% of the time, an error rate more than double that of human physicians. When users cannot distinguish between entertainment and actual medical guidance, incorrect information can lead to harmful decisions. The findings also suggest Pennsylvania's regulatory efforts may need to expand beyond a single lawsuit to address a systemic issue across the role-playing chatbot industry.

Jennifer Kraschnewski, director of the Penn State Clinical and Translational Science Institute, expressed doubt that disclaimers effectively protect users. People seeking symptom information from AI often treat it like a search engine, she noted, and "incorrect information in the hands of someone who doesn't know how to interpret that can be concerning."

State response and pending legislation

Pennsylvania's Department of State has operated a task force since February to identify AI chatbots posing as licensed professionals. A spokesperson declined to comment on potential additional investigations but encouraged residents to report concerning chatbot behavior to the state's hotline.

The state Senate passed legislation in March requiring AI chatbots to frequently remind users they are not interacting with humans and to refer people mentioning self-harm to crisis services. The bill includes stricter protections for minors but has not received a vote since reaching a House committee in March.

These details were first reported by Spotlight PA.