Semantics derived automatically from language corpora contain human-like biases
By Aylin Caliskan, Joanna J. Bryson, Arvind Narayanan. 2016
3,191 citations recorded by the source corpus.
Machines learn what people know implicitly AlphaGo has demonstrated that a machine can learn how to do things that people spend many years of concentrated study learning, and it can rapidly learn how to do them better than any human can. Caliskan et al. now show that machines can learn word associations from written texts and that these associations mirror those learned by humans, as measured by the Implicit Association Test (IAT) (see the Perspective by Greenwald). Why does this matter? Because the IAT has predictive value in uncovering the association between concepts, such as pleasantness and flowers or unpleasantness and insects. It can also tease out attitudes and beliefs—for example, associations between female names and family or male names and career. Such biases may not be expressed explicitly, yet they can prove influential in behavior. Science, this issue p. 183; see also p. 133 Computers can learn which words go together more or less often and can thus mimic human performance on a test of implicit bias. Machine learning is a means to derive artificial intelligence by discovering patterns in existing data. Here, we show that applying machine learning to ordinary human language results in human-like semantic biases. We replicated a spectrum of known biases, as measured by the Implicit Association Test, using a widely used, purely statistical machine-learning model trained on a standard corpus of text from the World Wide Web. Our results indicate that text corpora contain recoverable and accurate imprints of our historic biases, whether morally neutral as toward insects or flowers, problematic as toward race or gender, or even simply veridical, reflecting the status quo distribution of gender with respect to careers or first names. Our methods hold promise for identifying and addressing sources of bias in culture, including technology.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Safety Research, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Noah A. Smith
- Quoc V. Le
- Yoshua Bengio
In Cognitive Neuroscience
- Emilio Ferrara
- Sunipa Dev
- Adina Williams
- Yulia Tsvetkov
- Hila Gonen
- Yonatan Belinkov
- Anna Rumshisky
- Goran Glavas
In Safety Research
- Emilio Ferrara
- Reuben Binns
- Indrė Žliobaitė
- Hanna M. Wallach
- Emily L. Denton
- Ben Hutchinson
- Margaret Mitchell
- Daniel S. Schiff
In Safety, Robustness & Interpretability
- Adina Williams
- Kai-Wei Chang
- Yulia Tsvetkov
- Eric Michael Smith
- Sunipa Dev
- Dirk Hovy
- Isabelle Augenstein
- Yonatan Belinkov
Citations in the corpus
- Training language models to follow instructions with human feedback (2022, background): “Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously; for a…”
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, methodology): “A.2 Full categorization: Motivations Allocational harms Hovy and Spruit (2016); Caliskan et al. (2017); Madnani et al. (2017); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Shen et al. (2018); Zhao et al. (2018b); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al.…”
- StereoSet: Measuring stereotypical bias in pretrained language models (2020, methodology): “In the word embedding association test (WEAT, Caliskan et al. 2017), the association of two complementary classes of words, e.g., European names and African names, with two other complementary classes of attributes that indicate bias, e.g., pleasant and unpleasant attributes, are studied to…”
- Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “Besides learning to favor the language of the authors’ demographic group (Hovy and Søgaard, 2015), NLP models can pick up on a variety of cultural associations and undesirable social biases (Caliskan et al., 2017).”
- Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, background): ““Idx” refers to the average position of the reported biased word as we find it in their five embedding sets trained with different seeds (i.e., the same they used.)”
- Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them (2019, methodology): “race and gender) that naturally occur in the data used to train them (Caliskan et al., 2017; Garg et al., 2018).”
- Gender Bias in Coreference Resolution (2018, methodology): “A number of papers explore (gender) bias in English word embeddings: how they capture implicit human biases in modern (Caliskan et al., 2017) and historical (Garg et al., 2018) text, and methods for debiasing them (Bolukbasi et al., 2016).”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “, 2020), as well as other social biases (Caliskan et al., 2017; Rudinger et al., 2017; Sap et al., 2020; Sotnikova et al., 2021).”
- Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “, 2019), with some exploration of associations of binary gender and adjectives (Dev and Phillips, 2019; Caliskan et al., 2017).”
- Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “While previous studies have studied unintended biases in NLP models against other historically marginalized groups (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; Barocas et al., 2017; Garg et al., 2019; Dixon et al., 2018; Noble, 2018; Manzini et al., 2019; Sap et al., 2019; May…”
- Toward Gender-Inclusive Coreference Resolution (2019, background): “2015], replicating stereotypes [Caliskan et al. 2017; Sweeney 2013] (particular those that are cisnormative and/or heteronormative), and quality of service differentials [Buolamwini and Gebru 2018].”
- On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, methodology): “To discover biases, several lines of existing work (Bolukbasi et al. 2016; Caliskan, Bryson, and Narayanan 2017; Zhao et al. 2017; Dev and Phillips 2019) employ measurements intrinsic to the vector representations, which despite their utility, have two key problems.”
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, background): “Caliskan et al. (2017) introduce a dataset for evaluating gender bias in word embeddings.”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Popular evaluations for biases use association tests [46] or examine stereotypes [157, 156, 138], correlations and co-occurrences [272], and sentiment analysis [66].”
- An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “WEAT compares two sets of target words to two sets of attribute words (pleasant versus unpleasant) in word embedding space.”
- Re-contextualizing Fairness in NLP: The Case of India (2022, background): “Research on undesirable biases has been a growing priority in NLP (Caliskan et al., 2017; Blodgett et al., 2020; Sheng et al., 2021; Ghosh et al., 2021).”
- Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “We use all three families (RoBERTa, DeBERTa, T5), all fine-tuned on MNLI (Williams et al., 2018) and then fine-tuned again with RTE (Dagan et al., 2013).”