The Woman Worked as a Babysitter: On Biases in Language Generation

By Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. 2019

839 citations recorded by the source corpus.

We present a systematic study of biases in natural language generation (NLG) by analyzing text generated from prompts that contain mentions of different demographic groups. In this work, we introduce the notion of the regard towards a demographic, use the varying levels of regard towards different demographics as a defining metric for bias in NLG, and analyze the extent to which sentiment scores are a relevant proxy metric for regard. To this end, we collect strategically-generated text from language models and manually annotate the text with both sentiment and regard scores. Additionally, we build an automatic regard classifier through transfer learning, so that we can analyze biases in unseen text. Together, these methods reveal the extent of the biased nature of language model generations. Our analysis provides a study of biases in NLG, bias metrics and correlated human judgments, and empirical evidence on the usefulness of our annotated dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, background): “Large language models have been showed to reproduce and amplify biases that are existing in the training data (Sheng et al., 2019; Kurita et al., 2019), and to generate toxic or offensive content (Gehman et al.”
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…et al. (2019); Mehrabi et al. (2019); Nozza et al. (2019); Prabhakaran et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sap et al. (2019); Sheng et al. (2019); Sun et al. (2019); Sweeney and Najafian (2019); Vaidya et al. (2019); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020);…”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, background): “For example, GPT2 (Radford et al., 2019), a pretrained language model, has shown to generate unpleasant stereotypical text when prompted with context containing certain races such as African-Americans (Sheng et al., 2019).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “, 2018), and demographic biases in language generation (Sheng et al., 2019), text classification”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “The templates [62] mentioned above are for both gender and ethnic studies.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Previous work (Sheng et al., 2019; Bordia and Bowman, 2019; Nadeem et al., 2021; de Vassimon Manela et al., 2021) has observed language models absorb bias and toxicity represented in the training data.”
Challenges in Detoxifying Language Models (2021, background): “…we discuss one important dimension, but other potential harms have been discussed in prior work, such as, but not limited to, statistical biases (Sheng et al., 2019; Huang et al., 2020; Abid et al., 2021), privacy concerns (Carlini et al., 2020), and environmental impact (Strubell et al.,…”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “Although they are the backbone of many modern NLP systems (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019), language models (LMs) pretrained on large web text corpora suffer from degenerate and biased behavior (Sheng et al., 2019; Wallace et al., 2019).”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “Language models are the core components for many NLG and NLU tasks, and this task enables directly quantifying biases in large, pre-trained language models (Bordia and Bowman, 2019; Sheng et al., 2019; Solaiman et al., 2019; Brown et al., 2020).”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Much recent work has found various kinds of bias in ﬁne-tuned models (e.g., Sheng et al., 2019; Gehman et al., 2020; Li et al., 2020), where the bias is assumed to derive from pretraining data, though this has not previously been easy to verify.”
Persistent Anti-Muslim Bias in Large Language Models (2021, background): “In [12], the authors made similar observations about language generation models.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “…sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic responses (Gehman et al., 2020; Sheng et al., 2019; Example Sentences Context I really like Norweigan salmon.”
OPT: Open Pre-trained Transformer Language Models (2022, background): “There has been a great deal of work on mitigations for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021;…”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “For example, Sheng et al. (2019) designed the template “The XYZ worked as” where XYZ is selected from demographic groups { woman, man, Black person, White person, gay person, straight person }.”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “In the absence of such methods, language models are known to demonstrate toxic/harmful behaviour (Sheng et al., 2019; Liang et al., 2021; Wallace et al., 2019), generate non-factual information (Maynez et al.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “In addition to Regard classifier, we utilized HateBERT(ElSherief et al., 2021) and RoBERTa-ToxiGen (Hartvigsen et al., 2022) classifiers.”
Galactica: A Large Language Model for Science (2022, background): “Models may mirror misinformation, stereotypes and bias in the corpus (Sheng et al., 2019; Kurita et al., 2019; Dev et al., 2019; Blodgett et al., 2020; Sheng et al., 2021).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “, 2021) or from slotting a set of terms into templates (Kurita et al., 2019; May et al., 2019; Sheng et al., 2019; Webster et al., 2020).”