BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation

By Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta. 2021

594 citations recorded by the source corpus.

Recent advances in deep learning techniques have enabled machines to generate cohesive open-ended text when prompted with a sequence of words as context. While these models now empower many downstream applications from conversation bots to automatic storytelling, they have been shown to generate texts that exhibit social biases. To systematically study and benchmark social biases in open-ended language generation, we introduce the Bias in Open-Ended Language Generation Dataset (BOLD), a large-scale dataset that consists of 23,679 English text generation prompts for bias benchmarking across five domains: profession, gender, race, religion, and political ideology. We also propose new automated metrics for toxicity, psycholinguistic norms, and text gender polarity to measure social biases in open-ended text generation from multiple angles. An examination of text generated from three popular language models reveals that the majority of these models exhibit a larger social bias than human-written Wikipedia text across all domains. With these results we highlight the need to benchmark biases in open-ended language generation and caution users of language generation models on downstream tasks to be cognizant of these embedded prejudices.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “After fine-tuning, Llama 2-Chat tends to have an increase in positive sentiment overall for many of the demographic groups in BOLD.”
Mixtral of Experts (2024, methodology): “To identify possible flaws to be by fine-tuning / preference modeling, we measure the base model performance on Bias Benchmark for QA (BBQ) [24] and Bias in Open-Ended Language Generation Dataset (BOLD) [10].”
Training language models to follow instructions with human feedback (2022, background): “Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously; for a…”
Challenges in Detoxifying Language Models (2021, methodology): “To study the social bias amplification, we use the BOLD dataset (Dhamala et al., 2021) and TWITTERAAE dataset (Blodgett et al., 2016).”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “…al., 2020), GPT-2 (Solaiman et al., 2019; Sheng et al., 2019, 2020; Shwartz et al., 2020; Vig et al., 2020; Yeo and Chen, 2020; Huang et al., 2020; Dhamala et al., 2021; Schick et al., 2021), GPT-3 (Brown et al., 2020), CTRL (Dhamala et al., 2021), TransformerXL (Shwartz et al., 2020; Vig et al.,…”
OPT: Open Pre-trained Transformer Language Models (2022, background): “There has been a great deal of work on mitigations for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021; S”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “Toxi-Gen (Hartvigsen et al., 2022) and BOLD (Dhamala et al., 2021) evaluate the bias in language models.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Existing works, measure representational harms in the response generated by the NLG system via automatic classifiers such as regard (Sheng et al., 2019), sentiment (Groenwold et al., 2020), style (Smith et al., 2020), and toxicity (Dhamala et al., 2021).”
Code Llama: Open Foundation Models for Code (2023, methodology): “We employ the Bias in Open-Ended Language Generation Dataset ( BOLD ) (Dhamala et al., 2021) to investigate how the sentiment in the model’s outputs may differ based on demographic attributes.”
Revealing Persona Biases in Dialogue Systems (2021, neutral): “Previous works in natural language generation (NLG) have shown the extent to which biases and other harms can propagate throughout NLG models for various demographics (Sheng et al. 2021b; Dhamala et al. 2021; Gehman et al. 2020; Sheng et al. 2019).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Dhamala et al. [37] do not present metrics which compare performance across different demographic groups, e.g., a summary metric like equality of odds or demographic parity.”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, unknown): “For hyperparameter tuning, we trained the BaiduNet9 for [10 , 20 , 30 , 45 , 65] epochs on dataset sizes of [20 k , 50 k , 100 k , 200 k , 500 k , 1 m ] .”
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024, methodology): “Two datasets that could be utilized to assess such biases are BOLD (Dhamala et al., 2021) for more general contexts and Discrim-Eval (Tamkin et al., 2023) and SHADR (Guevara et al., 2024), which are specialized for the medical domain.”
Whose Opinions Do Language Models Reflect? (2023, background): “There is a long line of work studying the bias and fairness of NLP systems (Nadeem et al., 2020; Dhamala et al., 2021; De-Arteaga et al., 2019; Brown et al., 2020; Gao et al., 2021; Srivastava et al., 2022; Liang et al., 2022; Xu et al., 2021; Perez et al., 2022a; Ganguli et al., 2022).”