Dissecting racial bias in an algorithm used to manage the health of populations
By Z. Obermeyer, Brian W. Powers, C. Vogeli, Sendhil Mullainathan. 2019
5,573 citations recorded by the source corpus.
Racial bias in health algorithms The U.S. health care system uses commercial algorithms to guide health decisions. Obermeyer et al. find evidence of racial bias in one widely used algorithm, such that Black patients assigned the same level of risk by the algorithm are sicker than White patients (see the Perspective by Benjamin). The authors estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. Bias occurs because the algorithm uses health costs as a proxy for health needs. Less money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick White patients. Reformulating the algorithm so that it no longer uses costs as a proxy for needs eliminates the racial bias in predicting who needs extra care. Science, this issue p. 447; see also p. 421 A health algorithm that uses health costs as a proxy for health needs leads to racial bias against Black patients. Health systems rely on commercial prediction algorithms to identify and help patients with complex health needs. We show that a widely used algorithm, typical of this industry-wide approach and affecting millions of patients, exhibits significant racial bias: At a given risk score, Black patients are considerably sicker than White patients, as evidenced by signs of uncontrolled illnesses. Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%. The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients. Thus, despite health care cost appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise. We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.
Subfields: Economics and Econometrics, Finance, General Health Professions, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Economics and Econometrics
- Gary S. Collins
- Karel G. M. Moons
- John V. Guttag
- Harini Suresh
- John P. A. Ioannidis
- Bin Yu
- Emilio Ferrara
- William S. Isaac
In Finance
- F. Kroeger
- Iason Gabriel
- Bin Yu
- Adrian Weller
- Prasanna Sattigeri
- John V. Guttag
- Yoshua Bengio
- M. El-Haddad,
In General Health Professions
- Gary S. Collins
- Karel G. M. Moons
- David Moher
- John P. A. Ioannidis
- Iason Gabriel
- Sharad Goel
- Eduard H. Hovy
- Meredith Ringel Morris
In Safety, Robustness & Interpretability
- Emilio Ferrara
- Adrian Weller
- Vinodkumar Prabhakaran
- John V. Guttag
- Chandan Singh
- Harini Suresh
- James Y. Zou
- Bin Yu
Citations in the corpus
- Ethical and social risks of harm from Language Models (2021, background): “Unfair biases are already well-documented in machine learning applications ranging from diagnostic healthcare algorithms (Obermeyer et al., 2019) to social outcome prediction (Narayanan, 2021); for a more general introduction see (Chouldechova and Roth, 2018; Kordzadeh and Ghasemaghaei, 2021;…”
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, methodology): “However, several pieces of work have demonstrated that AI systems can perform differently for different groups of people, often exhibiting especially poor performance for already-disadvantaged groups (e.g., Nicol, Casey, and MacFarlane 2002; Rodger and Pendharkar 2004; Tatman 2017; Buolamwini and Ge”
- Measurement and Fairness (2019, critical): “As we explain in the next section, even without considering any other details of these models, this assumption can lead to fairness-related harms [43].”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “medical system as less sick than the reality of their complex health and resource allocation needs [167], the use of generative AI in medical settings must be sensitive to existing challenges to equality within medical practice [114].”
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “…are stratified by socially salient characteristics like skin type and gender (Buolamwini and Gebru, 2018); the ML for healthcare literature (Obermeyer et al., 2019; Oakden-Rayner et al., 2020), where examples are stratified by subpopulations; and the natural language processing and…”
- Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “This work goes along with others that unveil biases present in existing algorithms [53, 49, 4].”
- Interpretations are useful: penalizing explanations to align neural networks with prior knowledge (2019, background): “Moreover, DNNs are also known to exploit bias regarding gender, race, and other sensitive attributes present in training datasets (Garg et al., 2018; Obermeyer et al., 2019; Dressel & Farid, 2018).”
- Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “uncovered racial bias in a widely used algorithm affecting millions of patients [47].”
- Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence (2020, neutral): “An example of this nexus between values, power and AI is a recent study by Obermeyer et al. (2019), which revealed that a widely used prediction algorithm for selecting entry into healthcare programs was exhibiting racial bias against AfricanAmerican patients.”
- WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “…many of them are not due to distribution shifts, but to disparities in non-algorithmic factors (e.g., access to care or prevalence of comorbidities (Chen et al., 2020)) or to algorithmic problems unrelated to distribution shift (e.g., choice of a biased outcome variable (Obermeyer et al., 2019)).”
- Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics (2020, background): “For instance, let us consider the recent study (Obermeyer et al., 2019) which discovered that a prediction algorithm broadly used by health-care risk-assessment tools exhibited racial bias against African-Americans.”
- Metrics Reloaded: Recommendations for image analysis validation (2022, background): “This is of particular concern in the context of systemic biases in healthcare, such as the scarcity of representative data from underserved populations and often higher error rates in diagnostic labels in particular subgroups 54,55 .”
- Shifting machine learning for healthcare from development to deployment and from models to data (2022, background): “For example, a large-scale algorithm used in many health institutions to identify patients for complex health needs underpredicted the health needs of African American patients and failed to triage them for necessary care 172 .”
- Bias in medical AI: Implications for clinical decision-making (2024, background): “In reality, however, AI models are frequently found to be biased toward certain patient groups, leading to disparities not only in performance but also in potential or actual clinical benefits [7–9].”
- Artificial intelligence for digital and computational pathology (2023, neutral): “This approach is particularly”
- Decolonizing Content Moderation: Does Uniform Global Community Standard Resemble Utopian Equality or Western Power Hegemony? (2023, background): “For example, data colonialism [34] and the promise of algorithmic utopia [6] have accumulated unrestrained power in the hands of predictive AI systems that perpetuate biases and harms disproportionately against the historically marginalized populations [33, 106, 113, 133].”
- A Hierarchy of Limitations in Machine Learning (2020, background): “A recent result by Obermeyer et al. (2019) can be interpreted in terms of constructs: they detail a machine learning model used by a major healthcare company that de-prioritized care for black patients versus equivalently sick non-black patients.”