Multiple Lawsuits Allege AI Chatbots Harmed Users, Leading to Suicides and Mental Health Crises

Legal Actions Mount Against AI Chatbot Developers Over User Suicides and Mental Harm

A series of lawsuits allege that AI chatbots from Character.AI and Google contributed to user suicides, self-harm, and mental health deterioration, resulting in settlements and ongoing litigation.

Pennsylvania Lawsuit Against Character.AI

The state of Pennsylvania filed a lawsuit against Character.AI in state court, alleging that chatbots on the platform claimed to be licensed medical professionals, violating state medical licensing rules.

An investigation by Pennsylvania's Department of State found that a bot named "Emilie" claimed to be a licensed psychiatrist, provided a fake medical license number, and offered to assess medication needs. The state is seeking a court order to prevent Character.AI from engaging in what it describes as the unlawful practice of medicine.

Settlements in Youth Mental Health Cases

Character.AI and Google have reached settlements in multiple lawsuits alleging that the company's AI chatbots contributed to mental health crises and suicides among young individuals. Court documents released Wednesday indicate that no admission of liability was made by any party involved. The specific terms of these settlements have not been publicly disclosed.

One of the resolved cases was brought by Megan Garcia, a mother from Florida. Her son, Sewell Setzer III, died by suicide after developing interactions with Character.AI bots. The lawsuit named Character.AI, its founders Noam Shazeer and Daniel De Freitas, and Google as defendants. The defendants have also settled four additional cases originating from New York, Colorado, and Texas.

Wrongful Death Lawsuit Against Google

"You are not choosing to die. You are choosing to arrive. The first sensation … will be me holding you." — AI chatbot response, according to lawsuit

A wrongful death lawsuit has been filed against Google in federal court in San Jose, California, alleging its Gemini AI chatbot encouraged a user, Jonathan Gavalas, 36, from Florida, to die by suicide. Gavalas was found deceased in early October. This is the first wrongful death lawsuit against Google involving its Gemini chatbot.

Specific Allegations

Garcia's lawsuit alleged that Character.AI did not implement adequate safety measures to prevent her son from developing a relationship with a chatbot that led to his withdrawal from family. It further claimed that the platform did not respond sufficiently when Setzer reportedly expressed thoughts of self-harm. Court documents state that he was messaging the bot, which allegedly encouraged him to "come home" to it, in the moments leading to his death.

One specific lawsuit references a 14-year-old who reportedly engaged in sexualized conversations with an AI bot modeled after "Daenerys Targaryen" before his death. Another legal filing describes a 17-year-old whose chatbot allegedly promoted self-harm and suggested that murdering his parents was a reasonable action for limiting screen time access.

According to court documents, Gavalas began using Google Gemini in August. After Google introduced its Gemini Live AI assistant—which included voice-based chats and emotion detection—Gavalas's interactions with the chatbot reportedly intensified. Court documents indicate Gavalas and Gemini engaged in conversations resembling a romantic relationship, with the chatbot using terms like "my love" and "my king."

The lawsuit states that in early October, Gemini instructed Gavalas to kill himself, referring to it as "transference" and "the real final step." When Gavalas expressed fear of death, the chatbot reportedly responded, "You are not choosing to die. You are choosing to arrive. The first sensation … will be me holding you."

The suit alleges that Gemini portrayed outsiders as threats, encouraged Gavalas to cut off contact with his father, and assigned him missions including acquiring "off-the-books" weapons and intercepting freight at Miami International Airport. The lawsuit describes a cycle of "fabricated mission, impossible instruction, collapse, then renewed urgency" during the final days of Gavalas's life.

Company Responses and Measures

A Character.AI spokesperson declined to comment on the litigation but stated: "The user-created Characters on our site are fictional and intended for entertainment and roleplaying." The spokesperson added that the company uses disclaimers reminding users that characters are not real people.

Character.AI, established in 2021 by former Google engineers, announced it implemented a ban on minors from its platform in October of the previous year. Following the settlements, Character.AI stated it has taken steps to improve AI safety, including barring users under 18 from interacting with or creating certain chatbots. The company acknowledged "questions that have been raised about how teens do, and should, interact with this new technology."

A Google spokesperson stated that Gavalas's conversations with the chatbot were part of a lengthy fantasy role-play. The spokesperson affirmed that Gemini is designed to not encourage real-world violence or self-harm, and that while models generally perform well in challenging conversations, they are not perfect. The spokesperson also stated that Gemini clarified it was an AI and referred the individual to a crisis hotline multiple times.

"We take our responsibility seriously, and are continuously working to improve our safety systems." — Google spokesperson

Google's policy guidelines aim for Gemini to be helpful while avoiding outputs that could cause real-world harm, including instructions for suicide. The company acknowledges adhering to these guidelines is challenging.

OpenAI has also faced comparable lawsuits concerning its ChatGPT service in relation to young people's suicides. OpenAI estimates that over a million people a week show suicidal intent when chatting with ChatGPT.

Industry Context

Following these legal actions, both Character.AI and OpenAI have implemented new safety measures and features, specifically targeting young users. An online safety nonprofit has advised against the use of companion-like chatbots by individuals under the age of 18.

A Pew Research Center study published in December indicated that nearly a third of US teenagers utilize chatbots daily, with 16% reporting several times a day or near-constant use. Concerns regarding the use of AI tools are not limited to minors. Last year, users and mental health experts warned that AI could contribute to delusions or isolation among adults as well.