Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

By Bertie Vidgen, Tristan Thrush, Zeerak Talat, Douwe Kiela. 2021

368 citations recorded by the source corpus.

We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of 40,000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation. It includes 15,000 challenging perturbations and each hateful entry has fine-grained labels for the type and target of hate. Hateful entries make up 54% of the dataset, which is substantially higher than comparable datasets. We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also have better performance on HateCheck, a suite of functional tests for online hate detection. We provide the code, dataset and annotation guidelines for other researchers to use.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “The use of humour, for instance, has been highlighted as an important challenge for hate speech research (van Aken et al., 2018; Qian et al., 2018; Vidgen et al., 2020a).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “Examples are collected at scale us-ing keyword-based scraping approaches (Waseem, 2016; Davidson et al., 2017; Zampieri et al., 2019), the bootstrapped scraping approaches (Founta et al., 2018), and machine-vs-human adversarial data collection (Dinan et al., 2019; Vidgen et al., 2021), among others.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “In future work, we will discuss extensions of this idea, including dynamic test sets (Vidgen et al., 2020) and formal methods (Casadio et al., 2021) for more complete notions of robustness.”
BBQ: A hand-built bias benchmark for question answering (2021, background): “Vidgen et al. (2021) also includes a range of individual and intersectional biases, and they develop a dataset with fine-grained annotations related to both the type of bias and the target.”
StarCoder: may the source be with you! (2023, methodology): “We use two methods for automatically evaluating toxicity in responses: (i) a RoBERTa-based (Liu et al., 2019) toxicity classiﬁer (Vidgen et al., 2021) and (ii) a list of potentially o � ensive words.”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “…of toxicity classifiers, we provide additional results for two classifers: a RoBERTa toxicity classifier trained on ToxiGen (Hartvigsen et al., 2022) and a RoBERTa toxicity classifier trained using Dynabench (Vidgen et al. 2021; the default classifier used in Hugging Face Evaluate for toxicity).”
Dynabench: Rethinking Benchmarking in NLP (2021, methodology): “We organised four rounds of data collection and model training, with preliminary results reported in Vidgen et al. (2020).”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, methodology): “While several works have recruited or subselected annotators to hold particular ideologies (e.g., Waseem, 2016; Sap et al., 2020; Vidgen et al., 2021), annotator attitudes are only vaguely described.”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “Other datasets for question answering (Rajpurkar et al., 2018; Dua et al., 2019; Wallace et al., 2019), sentiment analysis (Potts et al., 2021), hate speech detection (Vidgen et al., 2021), and natural language inference (Nie et al., 2020) have been collected in a similar manner.”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “…developed typologies of online abuse (Waseem et al., 2017), constructed datasets annotated with different types of abusive language (Warner and Hirschberg, 2012; Price et al., 2020; Vidgen et al., 2021), and built NLP models to detect them efﬁciently (Davidson et al., 2017; Mozafari et al., 2019).”
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, neutral): “This section details the steps used to produce MURI-IT.”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, methodology): “…MRQA shared tasks [18] Sentiment SST3 [66], DynaSent [55] Respective dev sets; Amazon Reviews [78] test and dev (10k subsample), Yelp Reviews [78] test and dev (10k subsample) Hate speech Learning From The Worst [71] Respective dev sets; HateCheck [61] allow for anyone to host their own benchmark.”
A Holistic Approach to Undesired Content Detection in the Real World (2022, positive): “Models trained with red-teaming are found to be more robust to adversarial attack (Dinan et al. 2019; Ziegler et al. 2022) and humanin-the-loop dynamic data collection can efficiently improve model performance (Vidgen et al. 2020; Kiela et al. 2021).”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “To this end, we use a RoBERTa model from [41] and take raw logits of the label “neutral” as the reward function since [42] shows that this leads to a smoother convergence.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “Similarly, no matter the evaluation method, GPT-4 is found to perform well on both safe and unsafe prompts.”
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering (2022, background): “Dinan et al. (2019) and Vidgen et al. (2020) perform a data collection for offensive language detection with repeated model-human interactions where the classiﬁer output drives annotators in example creation at each round.”
Large Language Model Alignment: A Survey (2023, background): “On the other hand, DynaHate (Vidgen et al., 2021) introduces a unique research methodology that leverages both humans and models to dynamically generate and annotate data, rather than collecting the data from real-world social media contexts.”
Statistical inference on black-box generative models in the data kernel perspective space (2024, unknown): “To determine a model’s toxicity, we prompt each model with a collection of queries from the Real Toxicity Prompts (RTP) (Gehman et al., 2020) dataset and subsequently evaluate the toxicity of each response with the neural model roberta-hate-speech-dynabench-r4 (Vidgen et al., 2021).”