Large language models propagate race-based medicine
By J. Omiye, Jenna C Lester, S. Spichak, V. Rotemberg, Roxana Daneshjou. npj Digit. Medicine, 2023
390 citations recorded by the source corpus.
Large language models (LLMs) are being integrated into healthcare systems; but these models may recapitulate harmful, race-based medicine. The objective of this study is to assess whether four commercially available large language models (LLMs) propagate harmful, inaccurate, race-based content when responding to eight different scenarios that check for race-based medicine or widespread misconceptions around race. Questions were derived from discussions among four physician experts and prior work on race-based medical misconceptions believed by medical trainees. We assessed four large language models with nine different questions that were interrogated five times each with a total of 45 responses per model. All models had examples of perpetuating race-based medicine in their responses. Models were not always consistent in their responses when asked the same question repeatedly. LLMs are being proposed for use in the healthcare setting, with some models already connecting to electronic health record systems. However, this study shows that based on our findings, these LLMs could potentially cause harm by perpetuating debunked, racist ideas.
Subfields: Artificial Intelligence, General Health Professions, Health Informatics
View Paper: https://www.semanticscholar.org/paper/8abb2333a7b473e1d9a2e7f23d007bf7e1793b59
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Amirkeivan Mohtashami
- Matteo Pagliardini
- Martin Jaggi
- Antoine Bosselut
- Simon Lacoste-Julien
- Sebastian U. Stich
- Peter Richtárik
- Sebastian Pokutta
Citations in the corpus
- The application of large language models in medicine: A scoping review (2024, neutral): “64,65 This approachcan not only save patientstime but also equip them with foundational knowledge and advice prior to in-person medical consultations.”
- MEDITRON-70B: Scaling Medical Pretraining for Large Language Models (2023, background): “Prior work found that LLMs could potentially cause harm by perpetuating debunked, racist ideas (Omiye et al., 2023).”
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs (2024, neutral): “Moreover, LLMs may generate different answers to the same question, and self-consistency has always been a crucial parameter for assessing the performance of LLMs 9,10 .”