Reducing Gender Bias in Abusive Language Detection

By Ji Ho Park, Jamin Shin, Pascale Fung. 2018

385 citations recorded by the source corpus.

Abusive language detection models tend to have a problem of being biased toward identity words of a certain group of people because of imbalanced training datasets. For example, “You are a good woman” was considered “sexist” when trained on an existing dataset. Such model bias is an obstacle for models to be robust enough for practical use. In this work, we measure them on models trained with different datasets, while analyzing the effect of different pre-trained word embeddings and model architectures. We also experiment with three mitigation methods: (1) debiased word embeddings, (2) gender swap data augmentation, and (3) fine-tuning with a larger corpus. These methods can effectively reduce model bias by 90-98% and can be extended to correct model bias in other scenarios.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…(2017); Schnoebelen (2017); Blodgett et al. (2018); Curry and Rieser (2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Park et al. (2018); Shen et al. (2018); Thelwall (2018); Zhao et al. (2018b); Badjatiya et al. (2019); Bagdasaryan et al. (2019); Bamman et al.…”
Challenges in Detoxifying Language Models (2021, background): “This phenomenon has been linked to an increased relative prevalence of identity terms among toxic samples (Waseem and Hovy, 2016; Dixon et al., 2018; Park et al., 2018).”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “Corre-spondingly, models trained on such datasets have been shown to be overly sensitive to lexical features such as group identiﬁers (Park et al., 2018; Dixon et al., 2018; Kennedy et al., 2020), and to generalise poorly to other datasets (Nejadgholi and Kiritchenko, 2020; Samory et al., 2020).”
The Risk of Racial Bias in Hate Speech Detection (2019, methodology): “Since previous research has exposed the potential for other identity-based biases in offensive language detection (e.g., gender bias; Park et al., 2018), here we investigate racial bias against speech by African Americans, focusing on Twitter as it is a particularly important space for Black activis”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “…word embeddings (Bolukbasi et al., 2016; Brunet et al., 2018; Zhao et al., 2019), visual semantic role labeling (Zhao et al., 2017), natural language inference (He et al., 2019), abusive language classiﬁcation (Park et al., 2018 Table 1: Counts of gendered words in several dialogue datasets.”
Gender Bias in Contextualized Word Embeddings (2019, background): “…applications relying on automatic language analysis, including online news (Ross and Carter, 2011), advertisements (Sweeney, 2013), abusive language detection (Park et al., 2018), machine translation (Font and Costa-juss`a, 2019; Vanmassen-hove et al., 2018), and web search (Kay et al., 2015).”
Counterfactual Fairness in Text Classification through Robustness (2018, methodology): “Many methods have been proposed to address group fairness issues, such as re-calibrating score functions [5], ad-versarially learning fair representations [1, 14, 22], data rebalancing [2], and data augmentation using swaps of gender terms [18].”
Racial Bias in Hate Speech and Abusive Language Detection Datasets (2019, methodology): “Park et al. (2018) build upon this study, using templates to study gender differences in performance across two hate speech and abusive language detection datasets.”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, methodology): “Park et al. (2018) ﬁnd that gender-swapping was more effective at both removing bias and retaining performance than bias ﬁne-tuning.”
Assessing Social and Intersectional Biases in Contextualized Word Representations (2019, background): “Gender bias has also been demonstrated downstream on several applications of natural language processing, including sentiment analysis [18, 30], abusive language detection [23], image captioning [16] and text classification [11].”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “found that a dataset for toxicity classification contained a disproportionate association between words describing queer identities and text labeled as ‘‘toxic,’’ while Park et al.(24) found evidence of gender bias against women in similar datasets.”
Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification (2019, background): “ically have been shown to capture and reproduce biases common in society, for example mis-associating the names of frequently attacked identity groups (such as “gay”, and “muslim” etc.) with toxicity [5, 17]. This unintended model bias could be due to the demographic composition of the online user p”
Towards Controllable Biases in Language Generation (2020, background): “, 2019) and natural language understanding tasks (Rudinger et al., 2018; Zhao et al., 2018a; Park et al., 2018).”
Towards Debiasing Sentence Representations (2020, background): “In particular, Zhao et al. (2019), Park et al. (2018), and Garg et al. (2019) are not able to perform post-hoc debiasing and require changing the data or underlying word embeddings and retraining which is costly.”
Challenges in Automated Debiasing for Toxic Language Detection (2021, background): “tems has grown, several biases have been found in dataset and models, spurring various debiasing efforts to mitigate these individual biases (e.g., gender bias, racial bias; Park et al., 2018; Sap et al., 2019; Davidson et al., 2019).”