Generative AI Deciphers Social Determinants of Health in Doctor’s Notes

In a report published in npj Digital Medicine, researchers from Mass General Brigham outline the development of a generative artificial intelligence (AI) capable of identifying social determinants of health (SDoH) from doctor’s notes. Utilizing a large language model (LLM), this finely tuned tool successfully identified adverse SDoH in 93.8% of patients, a significant improvement compared to the mere two percent captured by diagnostic codes containing SDoH information. Importantly, the new model exhibited lower bias levels when contrasted with generalist models like GPT-4.

The researchers believe this innovative tool can contribute to the seamless identification of more patients who could benefit from additional health support resources and social work assistance. Furthermore, it sheds light on the often under-documented impact of SDoH on numerous patients.

Corresponding author Danielle Bitterman, MD, a faculty member in the Artificial Intelligence in Medicine (AIM) Program at Mass General Brigham, emphasized the clinical relevance of algorithms that can detect nuances in medical records, aiding doctors in providing better daily care for patients. Social determinants of health, encompassing factors such as a patient’s residence, employment, food security, and access to healthcare resources, are increasingly recognized for their influence on health and healthcare outcomes. Despite clinicians occasionally incorporating SDoH information in electronic health records (EHRs), the lack of systematic organization within the record poses a current challenge.

The development of the new generative AI tool by the Mass General Brigham team involved a meticulous process. They manually reviewed 800 clinician notes from 770 cancer patients who had undergone radiotherapy at the Department of Radiation Oncology at Brigham and Women’s. A predetermined list of six social determinants of health (SDoH), including employment status, housing, transportation, parental status, relationships, and the presence or absence of social support, was used to tag records containing references to one or more of these factors.

This dataset was then utilized to train the language models (LMs) to identify any references to SDoH in clinician notes. To test the newly developed model, the team used an additional 400 clinical notes from patients treated with immunotherapy at Dana-Farber Cancer Institute and from patients admitted to critical care units at Beth Israel Deaconess Medical Center.

The researchers discovered that fine-tuned LMs, especially Flan-T5 LMs, consistently identified rare references to SDoH in doctor’s notes. However, they observed that the learning capacity of the models was limited due to the scarcity of SDoH in the original training set, where only three percent of sentences in the notes contained any reference to an identified SDoH. To enhance performance, the team employed Chat GPT to generate an additional 900 synthetic examples of sentences indicating SDoH, using this as a second training set.

Recognizing the potential for bias in generative AI models developed for healthcare, the research team assessed their new tool’s performance against OpenAI’s GPT-4. They found that their tool was less likely to alter its identification of an SDoH based on a patient’s race/ethnicity or gender, indicating a reduced likelihood of perpetuating bias. Nevertheless, the researchers emphasized the need for ongoing research in this area to monitor and address algorithmic bias effectively.