Unintended Bias in Misogyny Detection

By Debora Nozza, Claudia Volpetti, Elisabetta Fersini. 2019

87 citations recorded by the source corpus.

During the last years, the phenomenon of hate against women increased exponentially especially in online environments such as microblogs. Although this alarming phenomenon has triggered many studies both from computational linguistic and machine learning points of view, less effort has been spent to analyze if those misogyny detection models are affected by an unintended bias. This can lead the models to associate unreasonably high misogynous scores to a non-misogynous text only because it contains certain terms, called identity, terms. This work is the first attempt to address the problem of measuring and mitigating unintended bias in machine learning models trained for the misogyny detection task. We propose a novel synthetic test set that can be used as evaluation framework for measuring the unintended bias and different mitigation strategies specific for this task. Moreover, we provide a misogyny detection model that demonstrate to obtain the best classification performance in the state-of-the-art. Experimental results on recently introduced bias metrics confirm the ability of the bias mitigation treatment to reduce the unintended bias of the proposed misogyny detection model. CCS CONCEPTS • Social and professional topics $\rightarrow$ Hate speech; • Computing methodologies $\rightarrow$ Neural networks.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…al. (2019); Fisher (2019); Font and Costa-jussà (2019); Garimella et al. (2019); Loukina et al. (2019); Mayfield et al. (2019); Mehrabi et al. (2019); Nozza et al. (2019); Prabhakaran et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sap et al. (2019); Sheng et al. (2019); Sun et al.…”
Cyberbullying Detection With Fairness Constraints (2020, background): “…model to retrieve informative numerical representations of the textual data (e.g. representations of tokens, words, sentences, or paragraphs) and training a deep neural network classiﬁer with these representations [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25].”
Pipelines for Social Bias Testing of Large Language Models (2022, methodology): “…out the an additional set of application-specific tests could be used to evaluate the models adapted for these tasks: for example, researchers could use hate speech check tests (Dixon et al., 2018; Nozza et al., 2019; Röttger et al., 2021) to verify social biases in hate speech detection models.”
Probing Toxic Content in Large Pre-Trained Language Models (2021, unknown): “…et al. (2020) who examine what words BERT generate in some ﬁll-in-the-blank experiments with regard to people with disabilities, and more recently Nozza et al. (2019) who assess hurtful auto-completion by multilingual PTLMs, we are not aware of other strategies designed to estimate toxic…”
Handling Bias in Toxic Speech Detection: A Survey (2022, background): “[59] identified 12 common female identity terms in the Evalita2018 [26] dataset.”
Theories of “Gender” in NLP Bias Research (2022, background): “[103] Debora Nozza, Claudia Volpetti, and Elisabetta Fersini.”
ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI (2021, background): “Misog-yny and sexism detection has been applied to social media in binary (Fersini et al., 2018; Nozza et al., 2019) and multi-class (Waseem and Hovy, 2016) settings.”
Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals (2022, background): “Research in hate speech detection considering gender and sexuality have mostly focus on sexism (Fersini et al., 2018; Basile et al., 2019; Nozza et al., 2019; Chiril et al., 2020; Fersini et al., 2020a,b; Attanasio and Pastor, 2020; Zein-29 ert et al., 2021; Mulki and Ghanem, 2021; Nozza, 2021;…”