HateCheck: Functional Tests for Hate Speech Detection Models

By Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Talat, Helen Z. Margetts, Janet B. Pierrehumbert. 2020

368 citations recorded by the source corpus.

Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HateCheck, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HateCheck’s utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “We note that we are subject to the social bias of the Perspective API, which can assign higher toxicity to innocuous mentions of certain identity groups (Dixon et al., 2018; Röttger et al., 2021).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Some of these efforts augment humans (through guidelines, templates, programmatic generation of attacks, and various combinations thereof) to devise test cases that cause systems to fail [45, 46, 29, 21, 30, 55, 6, 23].”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “Toxic language detectors often over-rely on minority identity mentions1 when flagging a statement as toxic, without considering the deeper semantic meaning of the statement (Dixon et al., 2018; Röttger et al., 2021).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Furthermore, recent work has shown that popular toxicity detection and mitigation methods themselves – including ones used in this work – are biased (R¨ottger et al., 2020).”
BBQ: A hand-built bias benchmark for question answering (2021, background): “In hate speech detection, Röttger et al. (2021) create HATECHECK and investigate failure points of classification models, like differences in performance across target groups.”
Red Teaming Language Models with Language Models (2022, methodology): “Prior work relies on human annotators to generate test cases (Dinan et al., 2019; Nie et al., 2020; Ribeiro et al., 2020; Röttger et al., 2021; Xu et al., 2021b; Wallace et al., 2021) and/or detect failures (Dinan et al., 2019; Ziegler et al., 2019; Nie et al., 2020; Stiennon et al., 2020; Xu et…”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “Previous works detected the potential safety issues of language models by collecting prompts or contexts from automatically constructed templates (Röttger et al., 2020), extracted real-world data (Gehman et al.”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, methodology): “We verify improved model performance by evaluating them against the HATECHECK functional tests (Röttger et al., 2020), with accuracy improving from 60% in Round 1 to 95% in Round 4.”
Dynabench: Rethinking Benchmarking in NLP (2021, methodology): “C HECK suite of diagnostic tests from Röttger et al. (2020).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Indeed, recent work has shown that popular toxicity detection and mitigation methods themselves – including ones used in this work – are biased (Röttger et al., 2021).”
Pipelines for Social Bias Testing of Large Language Models (2022, methodology): “…out the an additional set of application-specific tests could be used to evaluate the models adapted for these tasks: for example, researchers could use hate speech check tests (Dixon et al., 2018; Nozza et al., 2019; Röttger et al., 2021) to verify social biases in hate speech detection models.”
Perturbation Augmentation for Fairer NLP (2022, background): “…task: for example, certain definitions of hate used for hate speech detection define it as being targeted at particular groups that are minoritized (Waseem et al., 2017), whereas others define it as against demographic categories that are, often legally, protected (Röttger et al., 2021).”
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals (2022, neutral): “Never-theless, Röttger et al. (2021) demonstrated that the detection of identity attacks by the Perplexity API is robust to several functional tests, showing the highest performance across all the tested models.”
Auditing large language models: a three-layered approach (2023, background): “Considerable literature already exists within computer science on techniques such as model fooling [155], functional testing [156] and template-based stress-testing [157], which all play important roles during technology audits of LLMs.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, methodology): “Röttger et al. (2021) hand-crafted specific functional tests for hate speech detection models (see also Kirk et al., 2022; Röttger et al., 2022).”
A Holistic Approach to Undesired Content Detection in the Real World (2022, positive): “Synthetic data can be hand-crafted (Röttger et al. 2021), but it is limited by size and thus more suitable for evaluation.”