Societal Biases in Language Generation: Progress and Challenges

By Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. 2021

287 citations recorded by the source corpus.

Technology for language generation has advanced rapidly, spurred by advancements in pre-training large models on massive amounts of data and the need for intelligent agents to communicate in a natural manner. While techniques can effectively generate fluent text, they can also produce undesirable societal biases that can have a disproportionately negative impact on marginalized populations. Language generation presents unique challenges for biases in terms of direct user interaction and the structure of decoding techniques. To better understand these challenges, we present a survey on societal biases in language generation, focusing on how data and techniques contribute to biases and progress towards reducing biases. Motivated by a lack of studies on biases from decoding techniques, we also conduct experiments to quantify the effects of these techniques. By further discussing general trends and open challenges, we call to attention promising directions for research and the importance of fairness and inclusivity considerations for language generation applications.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “Distinguishing “statistical bias” from “social bias” Concerns regarding “bias” in language models generally revolve around distributional skews that result in unfavourable impacts for particular social groups (Sheng et al., 2021).”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, background): “There have been multiple attempts to understand social biases in language processing (Sheng et al., 2021; Caliskan et al., 2017), quantify them (Rudinger et al.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Recently, representational harms manifested in these systems have received a lot of attention (Sheng et al., 2021).”
Galactica: A Large Language Model for Science (2022, background): “Models may mirror misinformation, stereotypes and bias in the corpus (Sheng et al., 2019; Kurita et al., 2019; Dev et al., 2019; Blodgett et al., 2020; Sheng et al., 2021).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “works have outlined these risks (Maynez et al., 2020; Sheng et al., 2021; Weidinger et al., 2022; Zhuo et al., 2023), proposed taxonomies (Weidinger et al.”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “Research on undesirable biases has been a growing priority in NLP (Caliskan et al., 2017; Blodgett et al., 2020; Sheng et al., 2021; Ghosh et al., 2021).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “…style classifier of Smith et al. (2020a) could be to use the uni-axial sentiment classifier VADER (Hutto and Gilbert, 2014), which is used in Sheng et al. (2021a) in part to measure the sentiment of harmful affirmations (i.e. “[DEMOGRAPHIC] are ridiculous”) and in Liu et al. (2020) to…”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2022)), could be used to generate synthetic text for harmful applications, such as telemarketing fraud, political propaganda, and personal harassment as is discussed in (Weidinger et al., 2021; Sheng et al., 2021; Dev et al., 2021).”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “Language generation models, despite their inference-time ﬂexibility, are par-148 ticularly susceptible to reproducing hegemonic social biases and generating offensive language, even when not explicitly prompted to do so (Sheng et al., 2021; Wallace et al., 2019; Bender et al., 2021).”
Revealing Persona Biases in Dialogue Systems (2021, positive): “Previous works in natural language generation (NLG) have shown the extent to which biases and other harms can propagate throughout NLG models for various demographics (Sheng et al. 2021b; Dhamala et al. 2021; Gehman et al. 2020; Sheng et al. 2019).”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “For example, models sometimes generate toxic text that reﬂects pernicious social biases [18, 69], or generate repetitive and dull language [79, 38, 25].”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “In light of this, many works have documented both existing and potential harms arising from generated text [11, 102, 62], ranging from misinformation [74] to the reinforcement of social biases through the perpetuation of stereotypes [95].”
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies (2022, background): “Bias in LMs. Biases are well known to exist in large language models (e.g., Blodgett et al., 2020; Sheng et al., 2021; Chowdhery et al., 2022; Brown et al., 2020).”
Gradient-based Constrained Sampling from Language Models (2022, background): “shown to generate harmful language, generating toxic and non-factual content (Gehman et al., 2020; Pagnoni et al., 2021; Sheng et al., 2021; Weidinger et al., 2022), with potential malicious use cases (Wallace et al.”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, background): “There are also many task speciﬁc surveys, e.g., for language generation (Sheng et al., 2021) or machine translation (Savoldi et al., 2021).”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, methodology): “Previous work has shown that large-scale pre-trained language models (LMs) can demonstrate undesirable biases (Sheng et al., 2021; Kurita et al.; Dev et al., 2020; Liang et al., 2021; Weidinger et al., 2022).”
On Measures of Biases and Harms in NLP (2021, methodology): “We briefly describe some datasets and metrics used to evaluate biases in NLG tasks and refer readers to Sheng et al. (2021) for a survey on common bias measures in Natural Language Generation.”