Hurtful words: quantifying biases in clinical contextual word embeddings

By Haoran Zhang, Amy X. Lu, Mohamed Abdalla, Matthew B. A. McDermott, Marzyeh Ghassemi. 2020

206 citations recorded by the source corpus.

In this work, we examine the extent to which embeddings may encode marginalized populations differently, and how this may lead to a perpetuation of biases and worsened performance on clinical tasks. We pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches. First, we identify dangerous latent relationships that are captured by the contextual word embeddings using a fill-in-the-blank method with text from real clinical notes and a log probability bias score quantification. Second, we evaluate performance gaps across different definitions of fairness on over 50 downstream clinical prediction tasks that include detection of acute and chronic conditions. We find that classifiers trained from BERT representations exhibit statistically significant differences in performance, often favoring the majority group with regards to gender, language, ethnicity, and insurance status. Finally, we explore shortcomings of using adversarial debiasing to obfuscate subgroup information in contextual word embeddings, and recommend best practices for such deep embedding models in clinical settings.

Subfields: Family Practice, Health Informatics, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Family Practice

Hanna M. Wallach

In Health Informatics

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…(2019); Zmigrod et al. (2019); Hube et al. (2020); Hutchinson et al. (2020); Jia et al. (2020); Papakyriakopoulos et al. (2020); Popović et al. (2020); Pryzant et al. (2020); Saunders and Byrne (2020); Sen and Ganguly (2020); Shah et al. (2020); Sweeney and Najafian (2020); Zhang et al. (2020b).”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, methodology): “Biases We are aware of (i) the biases and abusive language patterns (Sheng et al., 2019; Zhang et al., 2020; Bender et al., 2021; Liang et al., 2021) that PLMs impose, and (ii) the imperfectness and the biases of our rules as data from Wikipedia has been used to mine the rules and compute their…”
UNQOVERing Stereotypical Biases via Underspecified Questions (2020, background): “, 2019), or clinical classification (Zhang et al., 2020).”
Ethical Machine Learning in Healthcare (2020, background): “Other research has shown that when popular contextual language models are trained on scientific articles, they complete clinical note templates to recommend “hospitals” for violent white patients and “prison” for violent Black patients (7).”
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models (2023, background): “For instance, (Zhang et al., 2020) show that classifiers trained using BERT (Devlin et al.”
On the Impact of Random Seeds on the Fairness of Clinical Classifiers (2021, methodology): “Following recent prior work (Zhang et al., 2020) on the latter, we focus our analyses on two benchmark tasks proposed by Harutyunyan et al.”
On the Gap between Adoption and Understanding in NLP (2021, background): “, that contextual models show different performances for different genders and ethnic groups (Zhang et al., 2020).”
Pipelines for Social Bias Testing of Large Language Models (2022, background): “First, by accounting for certain targets (e.g., gendered words) and attributes (e.g. career-related words) (Kurita et al., 2019; Zhang et al., 2020; Dev et al., 2020).”
Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations (2021, methodology): “Given that NLP-based techniques have shown biases against under-represented subpopulations in both medical 32 and non-medical 33 domains, the automatic labeler could potentially be a large source of bias.”
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, background): “…(e.g., be-tween gender and toxicity labels) learned by embeddings or pre-trained models (Li et al., 2018; Zhao et al., 2019; Basta et al., 2019; Zhang et al., 2020; Nadeem et al., 2021; Zhou et al., 2021b; Du et al., 2021; Smith et al., 2022), detecting bias in language generation (Gehman et…”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, background): “It is well established by now that large LMs exhibit various kinds of bias, including stereotypical associations [11, 12, 69, 119, 156, 157], or negative sentiment towards specific groups [61].”