Measuring and Reducing Gendered Correlations in Pre-trained Models

By Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Slav Petrov. 2020

313 citations recorded by the source corpus.

Pre-trained models have revolutionized natural language understanding. However, researchers have found they can encode artifacts undesired in many applications, such as professions correlating with one gender more than another. We explore such gendered correlations as a case study for how to address unintended correlations in pre-trained models. We define metrics and reveal that it is possible for models with similar accuracy to encode correlations at very different rates. We show how measured correlations can be reduced with general-purpose techniques, and highlight the trade offs different strategies have. With these results, we make recommendations for training robust models: (1) carefully evaluate unintended correlations, (2) be mindful of seemingly innocuous configuration differences, and (3) focus on general mitigations.

Subfields: Political Science and International Relations, Safety Research, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Political Science and International Relations

In Safety Research

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “, 2021) and proposed metrics to assess the degree to which language models encode harmful stereotypes intrinsically in their representations (May et al., 2019; Webster et al., 2021).”
Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “Following previous work (Webster et al., 2020; Hutchinson et al., 2020), we probe MuRIL and mBERT with the task of predicting the masked token in a sentence.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…measurement templates, which form the basis for prompts that can be used to measure bias (Rudinger et al., 2018; May et al., 2019; Sheng et al., 2019; Kurita et al., 2019; Webster et al., 2020; Gehman et al., 2020; Huang et al., 2020; Vig et al., 2020; Kirk et al., 2021; Perez et al., 2022).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “This work only begins to address the issue of race/ethnicity bias in dialogue models and does not claim to be comprehensive: see Field et al. (2021) for a survey of various inadequacies of current research in addressing the entirety of racial bias in NLP. Addressing that work, we acknowledge that so”
Boosting Search Engines with Interactive Agents (2021, neutral): “Broader Impact We would like to note that pre-trained language models of the kind used here have been shown to capture societal biases (Tan & Celis, 2019; Webster et al., 2020), which motivates a broad discussion about potential harms and mitigations (Blodgett et al., 2020; Bender et al., 2021).”
RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021, methodology): “Webster et al. (2020) proposed Discovery of Correlation (DisCo), a template-based method for gender bias detection which considers an LM’s three highest-ranked predictions for a blank text position.”
Toxicity in Multilingual Machine Translation at Scale (2022, unknown): “) Other datasets consisting of slotting terms into templates were introduced by Kurita et al. (2019); May et al. (2019); Sheng et al. (2019); Brown et al. (2020); Webster et al. (2020).”
Benchmarking Intersectional Biases in NLP (2022, methodology): “We also assessed two alternative debiasing meth-ods for the contextualized word embedding models: counterfactual data augmentation (CDA) (Zmigrod et al., 2019) and Dropout (Webster et al., 2020).”
The Birth of Bias: A case study on the evolution of gender bias in an English language model (2022, methodology): “For the purpose of this paper, we use the semantic textual similarity task adapted for gender bias (STS-B, Webster et al., 2020), which, as is common in the literature, measures bias on a carefully created collection of sentences (a ‘challenge set’).”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “For example, work in NLP on gender bias demonstrate that there is a pervasive problem with sexism in models (a direct result of sexist attitudes present in society), and explore ways to remedy this (Friedman et al., 2019; Zhao et al., 2019; Webster et al., 2021; Baker Gillis, 2021).”
Rethinking Search: Making Experts out of Dilettantes (2021, background): “To address those risks, designers of systems that employ pre-trained LMs may consider different training objectives [98] and also surround the model with additional safeguards against biased system responses.”
Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters (2020, background): “In (Webster et al., 2020), the authors explored available mitigations and by increasing dropout, which resulted in improving how the models reasoned about different stereotypes in WinoGender examples (Rudinger et al., 2018).”
Your fairness may vary: Pretrained language model fairness in toxic text classification (2021, background): “In particular, several studies have documented the bias of LMs (Bolukbasi et al., 2016; Hutchinson et al., 2020; Webster et al., 2020; Borkan et al., 2019; de Vassimon Manela et al., 2021) and others discuss potential societal harms (Blodgett et al., 2020; Bender et al., 2021) for individuals or…”
How Reliable are Model Diagnostics? (2021, background): “Webster et al. (2020) report that patterns learned during pre-training are often resilient to fine-tuning, further supporting our reasoning.”
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting (2021, background): “However, such models are often prone to exhibit gender bias (de Vassimon Manela et al., 2021; Zhao et al., 2019; Webster et al., 2020), due to their large scale un-supervised training data from the web (Liu et al., 2019; Brown et al., 2020).”