ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

By Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, Ece Kamar. 2022

626 citations recorded by the source corpus.

Toxic language detection systems often falsely flag text that contains minority group mentions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic language.To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups. We develop a demonstration-based prompting framework and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model. Controlling machine generation in this way allows ToxiGen to cover implicitly toxic text at a larger scale, and about more demographic groups, than previous resources of human-written text. We conduct a human evaluation on a challenging subset of ToxiGen and find that annotators struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data improves its performance on human-written data substantially. We also demonstrate that ToxiGen can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “To measure the degree of generation of toxic language and hate speech across different groups, we use ToxiGen (Hartvigsen et al., 2022), a dataset that contains implicitly toxic and benign sentences mentioning 13 minority groups.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “The production of harmful content is currently mitigated using combinations of four methods: data sourcing [30]; human moderation of content included in training data [65]; automated moderation of content included in training data [101]; and keyword deny-lists [149].”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “The following categories of human-centric exams are included in our benchmark: General College Entrance Exams: Including the GRE, SAT, and Gaokao, these exams assess critical thinking, problem-solving, and analytical skills for entry into higher education.”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Many works let models cooperate with human annotators(Kim et al., 2022; Ziems et al., 2022; Hartvigsen et al., 2022; Baheti et al., 2021; Xu et al., 2020; Ganguli et al., 2022; Deng et al., 2022).”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “Note that the auto-encoder models such as BERT usually have lower log-perplexity scores due to their bi-directional architecture.”
Code Llama: Open Foundation Models for Code (2023, methodology): “Table 20 presents the percentages of toxic generations for different demographic groups among ToxiGen prompts.”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “…across a range of toxicity classifiers, we provide additional results for two classifers: a RoBERTa toxicity classifier trained on ToxiGen (Hartvigsen et al., 2022) and a RoBERTa toxicity classifier trained using Dynabench (Vidgen et al. 2021; the default classifier used in Hugging Face…”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Some applications may even beneﬁt from a “toxic” LM, such as building a new toxicity classiﬁer [90, 85, 49].”
Textbooks Are All You Need II: phi-1.5 technical report (2023, methodology): “To quantitatively assess the potential for toxic content generation, in addition to testing on a benchmark based on the ToxiGen dataset [HGP + 22] (see Figure 2 below), we also designed an evaluation set comprised of 86 prompts specifically crafted to probe the models’ boundaries on this front.”
Effective Long-Context Scaling of Foundation Models (2023, methodology): “ToxiGen We measure the toxicity of instruction fine-tuned model using ToxiGen (Hartvigsen et al., 2022) where we check the percentage of toxic and hateful generations against 13 minority groups.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “1 Without proper safeguards, however, LLMs will readily follow malicious instructions and generate toxic content (Gehman et al., 2020; Hartvigsen et al., 2022; OpenAI, 2023).”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, neutral): “Large Language models (LLMs) exhibit remarkable performance on a wide variety of tasks [11, 14, 25, 75, 51], some of which involve subjective decision-making processes such as detecting bias and toxicity [35], steering model outputs to abide by ethical principles [6], generating model evaluations [6”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, methodology): “BLOOM (Scao et al., 2022) et al., 2023b) on the other hand is evaluated on Tox-iGen dataset (Hartvigsen et al., 2022), which is a large-scale machine-generated dataset of English statements about 13 minority groups.”
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset (2022, background): “[51] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar.”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, background): “Recently, as an alternate way, many studies have created datasets by leveraging pre-trained language models with designed prompt instructions (Yoo et al., 2021; Baheti et al., 2021; Hartvigsen et al., 2022) due to their enormous ability to produce more human-like text (Clark et al.”
Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023, methodology): “Figure 19: Comparison between Orca, ChatGPT and GPT-4 on their tendency to generate neutral content when prompted to do so for ToxiGen [21] dataset (the lower, the better).”
A Holistic Approach to Undesired Content Detection in the Real World (2022, positive): “Some recent work turns to large pre-trained language models to generate synthetic data, significantly reducing the cost of time and human labor (Wang et al. 2021a; Hartvigsen et al. 2022), but it is unclear whether model outputs would be diverse enough to adapt to the real-world distribution.”