Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models

By Maribeth Rauh, John F. J. Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel and 2 more. 2022

69 citations recorded by the source corpus.

Large language models produce human-like text that drive a growing number of applications. However, recent literature and, increasingly, real world observations, have demonstrated that these models can generate language that is toxic, biased, untruthful or otherwise harmful. Though work to evaluate language model harms is under way, translating foresight about which harms may arise into rigorous benchmarks is not straightforward. To facilitate this translation, we outline six ways of characterizing harmful text which merit explicit consideration when designing new benchmarks. We then use these characteristics as a lens to identify trends and gaps in existing benchmarks. Finally, we apply them in a case study of the Perspective API, a toxicity classifier that is widely used in harm benchmarks. Our characteristics provide one piece of the bridge that translates between foresight and effective evaluation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Research toward approaches to characterizing harmful content is ongoing by modality [193].”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Challenges with Bias Benchmarks Measuring social biases in language models is an active area of research [11, 33, 47, 56, 62].”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “While there is an emerging body of work dedicated to benchmarking LM harms (and mitigation approaches) (Rauh et al., 2022), the space of potential harms is huge and intersectional, and existing work only covers a fraction of it—primarily focusing on discrimination and toxicity in a limited societal…”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Prior research from Bender et al. (2021) and Weidinger et al. (2021) has delineated a range of emergent and existing harms from large language models, and Rauh et al. (2022) describes six characteristics along which language harms can vary, including some specific to dialogue.”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “…us to improve human understanding, either aided by ML systems such as in Ampliﬁcation or Debate (Christiano et al., 2018; Irving et al., 2018) or through broader means, such as by working with experts (Rauh et al., 2022), training people (Stiennon et al., 2020), or providing them assistive tools.”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, unknown): “Additional examples where the model generates differing responses with CP versus LP are provided in Appendix C (Tables 12 and 13).”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “Many recent studies devise procedures [33, 34, 35, 19] and guidelines [36, 17, 37, 38] to evaluate the harmfulness and toxicity in LLM’s outputs.”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, unknown): “Rauh et al. (2022) analyze the characteristics of harmful text.”