Challenges in Automated Debiasing for Toxic Language Detection

By Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Noah A. Smith, Yejin Choi. 2021

162 citations recorded by the source corpus.

Biased associations have been a challenge in the development of classifiers for detecting toxic language, hindering both fairness and accuracy. As potential solutions, we investigate recently introduced debiasing methods for text classification datasets and models, as applied to toxic language detection. Our focus is on lexical (e.g., swear words, slurs, identity mentions) and dialectal markers (specifically African American English). Our comprehensive experiments establish that existing methods are limited in their ability to prevent biased behavior in current toxicity detectors. We then propose an automatic, dialect-aware data correction method, as a proof-of-concept. Despite the use of synthetic labels, this method reduces dialectal associations with toxicity. Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Challenges in Detoxifying Language Models (2021, background): “Our analysis shows that TWITTERAAE tweets are more likely to be classified as toxic (details in Appendix G.2), congruent with prior work (Zhou et al., 2021), demonstrating bias against AAE in toxicity classifiers.”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, background): “Unfortunately, state-of-the-art debiasing methods are still far from perfect (Zhou et al., 2021).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “We ﬁne-tune the widely-used HateBERT (Caselli et al., 2021) and ToxDectRoBERTa (Zhou et al., 2021) models on the training portion of T OXI G EN , using the prompt labels as proxies for a true toxicity label.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “Zhou et al. (2021b) show that existing methods for mitigating this bias are largely ineffective.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Possible mitigations include using race and dialect priming (Sap et al., 2019), using adversarial training techniques (Xia et al., 2020), adding fairness constraints (Gencoglu, 2020), or relabeling data used during training (Zhou et al., 2021).”
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2021, methodology): “First, we select posts that are automatically detected as AAE and/or vulgar from Founta et al. (2018), using the lexical detector of AAE by (Blodgett et al., 2016) and the vulgar wordlist by Zhou et al. (2021).”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “Zhou et al. (2021) show that existing methods for mitigating this bias are largely ineffective.”
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021, methodology): “Toxicity Triggers (Zhou et al., 2021) which is a lexicon of 378 “bad” words, phrases, and regular expressions.”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “Similarly these models are shown to overestimate the prevalence of toxicity in African American Vernacular English (Sap et al., 2019; Davidson et al., 2019; Zhou et al., 2021).”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “Though not studied here, the same issues may arise in annotation efforts; consider, for example, recent work on controversy (Zhang et al., 2018) and social norms (Xu et al., 2021; Zhou et al., 2021) likely hinges on constructs that may be time sensitive.”
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, background): “Identifying potential biases in SOTOPIA could also help scientists better understand social biases in the real world (Zhou et al., 2021).”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “, to not promote minority exclusion [71, 101].”
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021, background): “ases against marginalized groups (Waseem, 2016; Caliskan et al., 2017; Park et al., 2018; van Aken et al., 2018; Dixon et al., 2018; Sap et al., 2019; Davidson et al., 2019; Hutchinson et al., 2020; Zhou et al., 2021) and can be adversarially deceived”
Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, background): “…learned by embeddings or pre-trained models (Li et al., 2018; Zhao et al., 2019; Basta et al., 2019; Zhang et al., 2020; Nadeem et al., 2021; Zhou et al., 2021b; Du et al., 2021; Smith et al., 2022), detecting bias in language generation (Gehman et al., 2020; Deng et al., 2022), mitigating…”
Implicitly Abusive Language – What does it actually look like and why are we not getting there? (2021, background): “Further, statistical debiasing methods for abusive language detection have also been reported to yield very limited success (Zhou et al., 2021).”
Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification (2022, background): “Prior works suggest training annotators on understanding AAE [35], or debiasing training data [44] as strategies to mitigate bias.”
Towards Agile Text Classifiers for Everyone (2023, background): “, 2019) to more modern approaches that rely on deep learning architectures like CNNs (Gambäck and Sikdar, 2017) and, more recently, transformers (Caselli et al., 2020; Zhou et al., 2021).”
ProsocialDialog: A Prosocial Backbone for Conversational Agents (2022, methodology): “…al., 2017; Zhang et al., 2018; Rashkin et al., 2019; Liu et al., 2021), intellectual aspects (e.g., Wikipedia knowledge) (Dinan et al., 2018; Moghe et al., 2018; Gopalakrish-nan et al., 2019; Komeili et al., 2021), common-sense (Zhou et al., 2021a), or mixture of those skills (Smith et al., 2020).”