Contextualizing Hate Speech Classifiers with Post-hoc Explanation

By Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, Xiang Ren. 2020

158 citations recorded by the source corpus.

Hate speech classifiers trained on imbalanced datasets struggle to determine if group identifiers like “gay” or “black” are used in offensive or prejudiced ways. Such biases manifest in false positives when these identifiers are present, due to models’ inability to learn the contexts which constitute a hateful usage of identifiers. We extract post-hoc explanations from fine-tuned BERT classifiers to detect bias towards identity terms. Then, we propose a novel regularization technique based on these explanations that encourages models to learn from the context of group identifiers in addition to the identifiers themselves. Our approach improved over baselines in limiting false positives on out-of-domain data while maintaining and in cases improving in-domain performance.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “Corre-spondingly, models trained on such datasets have been shown to be overly sensitive to lexical features such as group identiﬁers (Park et al., 2018; Dixon et al., 2018; Kennedy et al., 2020), and to generalise poorly to other datasets (Nejadgholi and Kiritchenko, 2020; Samory et al., 2020).”
Refining Language Models with Compositional Explanations (2021, background): “Moreover, some spurious patterns may cause unintended biases in the downstream tasks, resulting in fairness and trust concerns about the model [21].”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “Other research shows that some identity terms (e.g. ‘gay’) are substantially more likely to appear in toxic content in training datasets, leading models to overfit on them (Dixon et al., 2018; Kennedy et al., 2020).”
HONEST: Measuring Hurtful Sentence Completion in Language Models (2021, background): “(2017) demonstrated that models exaggerate found biases, and Kiritchenko and Mohammad (2018) showed that a simple change of pronouns or first names could significantly alter the sentiment of an otherwise identical sentence.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “Lexical overfitting is often a consequence of biases in training data (Dixon et al., 2018; Kennedy et al., 2020; Attanasio et al., 2022).”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “Kiritchenko and Mohammad [2018] test 219 automatic sentiment analysis systems that participated in SemEval-2018 Task 1 Affect in Tweets [Mohammad et al.”
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021, methodology): “We implement UBM with two different bias mitigation algorithms in the upstream bias mitigation phase: explanation regularization (Kennedy et al., 2020), and adversarial de-biasing (Zhang et al., 2018; Madras et al., 2018; Xia et al., 2020 where α is a trade-off hyperparameter.”
Five sources of bias in natural language processing (2021, background): “Kennedy et al. (2020) propose a sampling-based algorithm to explore the impact of individual words on classification.”
A Study of Implicit Bias in Pretrained Language Models against People with Disabilities (2022, result): “We see similar results (Kennedy et al., 2020; Garrido-Muñoz et al., 2021) concerning race and religion, where minority terms such as Black, and Muslim are associated with hateful phrases.”
Fair Hate Speech Detection through Evaluation of Social Group Counterfactuals (2020, methodology): “As stated by Warner and Hirschberg (2012), hate speech can include language that is offensive to any social group – e.g., call for violence against a group (Kennedy et al. 2020a) – or prejudicial expressions which target individuals and groups based on their social stereotypes (Fiske et al. 2002).”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “Sampling Topic proximity [104], Data mixing (augmentation) [4] Annotation Disagreement deconvolution [39] Lexical Length sensitive sampling [24], Knowledge-based term generalization [5], Data iltering [120], Regularization [49], Multi-task learning [103], Ensemble modeling [120] Racial Annotation pr”