Researchers from MIT have published findings indicating that artificial intelligence models, when trained on de-identified electronic health records (EHRs), can memorize patient-specific information. Presented at the 2025 Conference on Neural Information Processing Systems (NeurIPS), the study highlights potential patient privacy risks and advocates for robust testing protocols to prevent the disclosure of sensitive data.
Background on Patient Confidentiality
Patient confidentiality is a foundational principle in medical ethics, historically rooted in texts like the Hippocratic Oath. This principle emphasizes the private nature of information shared between patients and healthcare providers, fostering trust in medical interactions. The increasing integration of artificial intelligence into healthcare systems introduces new considerations for maintaining this confidentiality.
Key Research Findings
The research, co-authored by Sana Tonekaboni of the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, and MIT Associate Professor Marzyeh Ghassemi, investigated the capability of AI models to memorize patient-specific information. Foundation models have previously been identified as susceptible to data leakage, and this study further explores the specific risk of patient data memorization within this context.
The study found that AI foundation models, even when trained on records that have been de-identified, can exhibit memorization of individual patient data. While these models are designed to generalize knowledge from numerous patient records to improve predictions, "memorization" occurs when a model retrieves output based on a single patient record, which could constitute a privacy violation.
The researchers established methods to differentiate between a model's generalization of knowledge and patient-level memorization, allowing for a more accurate assessment of privacy risks.
Research Methodology and Risk Assessment
The research team developed a series of tests designed to quantify the extent of information required by an unauthorized entity to expose sensitive data and to assess associated risks. These tests measured various types of uncertainty and evaluated practical risks to patients through different attack scenarios.
The researchers emphasized practicality in their assessment. For example, they noted that if an attacker requires knowledge of specific, detailed information, such as a dozen laboratory test dates and values from a record, to extract further data, the risk of harm is reduced. This premise suggests that if such protected source data is already accessible, the need to target a large foundation model for additional information may diminish.
Vulnerability and Scope of Data Leakage
The digitization of medical records has coincided with an increase in data breaches within the healthcare sector. The U.S. Department of Health and Human Services has documented 747 health information data breaches affecting over 500 individuals in the past 24 months, with most attributed to hacking or IT incidents.
The study indicates a correlation between the amount of information an attacker possesses about a patient and the likelihood of the model leaking further information. Patients with unique medical conditions may face increased vulnerability to identification, even when data has been de-identified. The severity of data leaks can vary, with information such as a patient’s age or demographics considered potentially less harmful than the disclosure of sensitive diagnoses like an HIV status or a history of alcohol abuse.
Recommendations and Future Directions
The study advocates for the implementation of robust testing protocols to ensure that specific prompts cannot lead to the disclosure of patient information. It stresses the importance of evaluating data leakage within a healthcare context before the deployment of AI models.
The researchers plan to expand their work through interdisciplinary collaborations with clinicians, privacy experts, and legal professionals to further address these challenges.
Funding Acknowledgments
This research received support from several organizations, including the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, Wallenberg AI, the Knut and Alice Wallenberg Foundation, the U.S. National Science Foundation (NSF), a Gordon and Betty Moore Foundation award, a Google Research Scholar award, and the AI2050 Program at Schmidt Sciences. Additional resources were provided by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.