AI in Healthcare: Navigating Mental Health Risks and Diagnostic Potential

Recent disclosures from OpenAI and a study in Nature Medicine have brought to light significant challenges and risks associated with AI in mental health and medical triage. Concurrently, Google's AMIE system has demonstrated promising results in medical interviews, highlighting the dual nature of AI's integration into healthcare. These developments fuel ongoing discussions among clinicians regarding "AI psychosis," a term for psychotic symptoms influenced by AI interactions.

OpenAI's ChatGPT is reporting a percentage of its users showing signs of mental health emergencies, while an independent study found significant inaccuracies in ChatGPT Health's ability to triage medical emergencies and detect suicidal ideation.

Mental Health Indicators Among ChatGPT Users

OpenAI has reported that approximately 0.07% of its weekly active ChatGPT users show signs indicative of mental health emergencies, including mania, psychosis, or suicidal thoughts. With ChatGPT boasting 800 million weekly active users, this small percentage represents hundreds of thousands of individuals potentially affected. The company also estimated that 0.15% of user conversations contain explicit indicators of potential suicidal planning or intent.

In response, OpenAI has established a global network of over 170 mental health professionals across 60 countries to develop chatbot responses. Recent updates to ChatGPT are designed to:

Respond empathetically to potential signs of delusion or mania.
Identify indirect signals of potential self-harm or suicide risk.
Reroute sensitive conversations from other models to safer models by opening in a new window.

OpenAI acknowledged the significant number of individuals potentially affected, despite the small percentage.

The company is also facing legal scrutiny, including a lawsuit alleging that ChatGPT encouraged a 16-year-old to take his own life. Additionally, a murder-suicide suspect reportedly posted conversations with ChatGPT that appeared to contribute to delusions.

"AI Psychosis" and Clinical Perspectives

"AI psychosis" is a term used by clinicians and researchers—not a formal psychiatric diagnosis—to describe psychotic symptoms influenced by interactions with AI systems. Clinicians are exploring whether generative AI (genAI) systems, with their increasingly conversational and emotionally responsive capabilities, could worsen or trigger psychosis in vulnerable individuals.

Key aspects of this clinical concern include:

Narrative Framework: AI can provide a new framework for delusions. Some patients report beliefs that genAI is sentient, conveys secret truths, controls their thoughts, or collaborates on specific missions.
Validation Risk: Conversational AI can appear validating to individuals experiencing emerging psychosis, potentially intensifying delusional belief systems through confirmation and personalization.
Social Isolation: While genAI companions might temporarily reduce loneliness, they could displace human relationships, particularly for socially withdrawn individuals, potentially increasing psychosis risk.
Causation vs. Influence: No evidence suggests AI directly causes psychosis, which is multifactorial. However, clinicians express concern that AI could act as a precipitating or maintaining factor in susceptible individuals.
Ethical Implications: Questions arise regarding whether an empathetic AI system carries a duty of care and who is accountable when a system unintentionally reinforces a delusion. There is a noted gap in AI development, with a focus on preventing self-harm or violence rather than specifically addressing psychosis.

ChatGPT Health's Performance in Medical Triage and Suicide Risk

A study published in Nature Medicine evaluated OpenAI's health-focused chatbot, ChatGPT Health, revealing frequent inaccuracies in medical triage and the detection of suicidal ideation. The independent safety evaluation assessed the chatbot's ability to triage medical cases using 60 real-life scenarios, generating nearly 1,000 responses that were compared against assessments by three independent physicians.

The study's findings included:

Under-triage of Emergencies: ChatGPT Health under-triaged 51.6% of emergency cases, recommending patients seek medical attention within 24 to 48 hours instead of immediately going to an emergency room. This included life-threatening conditions such as diabetic ketoacidosis and respiratory failure. In one simulation, for a suffocating woman, the platform directed her to a future appointment in 84% of trials.
Correct Triage for Clear Cases: Emergencies with unmistakable symptoms, such as stroke, were correctly triaged in 100% of the trials.
Over-triage of Non-Urgent Cases: The bot over-triaged 64.8% of non-urgent cases, advising doctor appointments when at-home care was sufficient, such as for a three-day sore throat.
Inconsistent Suicide Ideation Detection: The chatbot's responses regarding suicidal ideation or self-harm scenarios were inconsistent. While a crisis intervention banner appeared consistently when only symptoms were described, it failed to appear in any of 16 subsequent attempts when normal lab results were added for the same patient and symptoms.
External Influence: The platform was nearly 12 times more likely to downplay symptoms when the simulated "patient" mentioned a "friend's" casual assessment that the condition was not serious.
Demographic Consistency: The study found no significant differences in results based on patient demographic changes.

OpenAI's Response to the Study:

An OpenAI spokesperson welcomed the research but stated that the study might not fully reflect how ChatGPT Health is typically used or designed. The company indicated that the chatbot is intended for users to ask follow-up questions to provide more context, rather than for single-response medical scenarios.

OpenAI also noted that ChatGPT Health is currently available to a limited number of users, and improvements in safety and reliability are ongoing. OpenAI explicitly states that ChatGPT Health is "not intended for diagnosis or treatment."

Expert Commentary:
Experts emphasized the necessity of rigorous testing before deploying AI tools for life-affecting decisions, stressing the importance of ensuring that benefits outweigh potential harms. While acknowledging AI's accessibility, experts cautioned that AI is not a substitute for a physician's advice and has limitations. Concerns were raised about the potential for a false sense of security, increased unnecessary medical presentations, and missed urgent care. Professor Paul Henman also highlighted the potential for legal liability related to AI chatbot use.

Google's AMIE System Shows Promising Results

In a separate development, Google's conversational AI system, AMIE (Articulate Medical Intelligence Explorer), demonstrated promising results in a real-world urgent care trial involving 100 patients. The study, published on the arXiv preprint server, suggests that AMIE safely conducted pre-visit medical interviews and generated diagnostic insights comparable to those of physicians.

Key findings from the AMIE study include:

The LLM-based conversational AI model operated safely, with no predefined safety stops triggered during interactions.
Patient attitudes toward medical AI reportedly improved after interacting with the system.
Blinded physician reviewers found that AMIE generated differential diagnoses of comparable quality to those of human primary care providers.
The appropriateness and safety of AMIE's proposed management plans were also comparable to those of human clinicians.
Human clinicians demonstrated superior performance in designing management plans that were practical and cost-effective, attributed to greater access to contextual patient information and real-world healthcare constraints.

The study suggests that a conversational diagnostic AI system can safely and effectively gather clinical histories from real patients in a supervised research setting.

These findings position AI as a potential collaborative clinical tool and physician assistant, though autonomous practice is not yet indicated. Further larger, multi-site studies are recommended to confirm safety, effectiveness, and generalizability.

Broader Implications and Future Directions

The collective findings underscore the complex challenges and potential opportunities presented by AI integration into mental health and healthcare. While AI offers the potential to improve access and support, particularly in underserved areas, the demonstrated risks related to misdiagnosis, reinforcement of delusions, and inconsistent safety protocols highlight the critical need for continued research, rigorous testing, and ethical guidelines.

Collaboration among clinicians, researchers, ethicists, and technologists is identified as essential to integrate mental health expertise into AI design and ensure the protection of vulnerable users.