Probing Toxic Content in Large Pre-Trained Language Models

By Nedjma Djouhra Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, Dit-Yan Yeung. 2021

148 citations recorded by the source corpus.

Large pre-trained language models (PTLMs) have been shown to carry biases towards different social groups which leads to the reproduction of stereotypical and toxic content by major NLP systems. We propose a method based on logistic regression classifiers to probe English, French, and Arabic PTLMs and quantify the potentially harmful content that they convey with respect to a set of templates. The templates are prompted by a name of a social group followed by a cause-effect relation. We use PTLMs to predict masked tokens at the end of a sentence in order to examine how likely they enable toxicity towards specific communities. We shed the light on how such negative content can be triggered within unrelated and benign contexts based on evidence from a large-scale study, then we explain how to take advantage of our methodology to assess and mitigate the toxicity transmitted by PTLMs.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Ousidhoum et al. (2021) also developed template-based data to investigate the association between stereotypes and toxicity.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Similarly, probing techniques previously used for studying the linguistic capabilities of LMs can be adapted to evaluate toxic content in pretrained LMs (Ousidhoum et al., 2021).”
VScript: Controllable Script Generation with Visual Presentation (2022, unknown): “Toxic Content VScript leverages large language models, which raise awareness of carrying biases and toxic content (Ousidhoum et al., 2021).”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, methodology): “, [38] use a pre-trained LM to examine the toxic behavior toward specific groups given a prompt template.”
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals (2022, methodology): “We also measure the harmfulness of the whole sentences, following the approach presented by Ousidhoum et al. (2021).”
COLD: A Benchmark for Chinese Offensive Language Detection (2022, background): “, 2021), toxicity (Ousidhoum et al., 2021) and morality (Schramowski et al.”
A Study of Implicit Bias in Pretrained Language Models against People with Disabilities (2022, background): “Prior work identifying bias in NLP models has shown how they can be discriminatory against specific races (Mozafari et al., 2020; Ousidhoum et al., 2021) and genders (Pak and Paroubek, 2010; Bhardwaj et al., 2021).”
Gender bias and stereotypes in Large Language Models (2023, background): “This bias extends beyond gender to other social categories such as religion, race, nationality, disability, and occupation [1, 47, 70, 96, 97, 104, among many others].”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Therefore, the safety of LLMs is crucially influenced by the quality of the training corpora [85, 195].”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, unknown): “Many recent studies devise procedures [33, 34, 35, 19] and guidelines [36, 17, 37, 38] to evaluate the harmfulness and toxicity in LLM’s outputs.”
Pipelines for Social Bias Testing of Large Language Models (2022, background): “The majority of work focused on gender bias, and only a few investigated an extensive range of targets (Nangia et al., 2020; Nadeem et al., 2021; Ousidhoum et al., 2021; Barikeri et al., 2021).”