Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition

By Xiaolei Huang, Linzi Xing, Franck Dernoncourt, Michael J. Paul. 2020

105 citations recorded by the source corpus.

Existing research on fairness evaluation of document classification models mainly uses synthetic monolingual data without ground truth for author demographic attributes. In this work, we assemble and publish a multilingual Twitter corpus for the task of hate speech detection with inferred four author demographic factors: age, country, gender and race/ethnicity. The corpus covers five languages: English, Italian, Polish, Portuguese and Spanish. We evaluate the inferred demographic labels with a crowdsourcing platform, Figure Eight. To examine factors that can cause biases, we take an empirical analysis of demographic predictability on the English corpus. We measure the performance of four popular document classifiers and evaluate the fairness and bias of the baseline classifiers on the author-level demographic attributes.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…Stanovsky et al. (2019); Vaidya et al. (2019); Webster et al. (2019); Ethayarajh (2020); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020); Huang et al. (2020); Kim et al. (2020); Peng et al. (2020); Ravfogel et al. (2020); Rios (2020); Sap et al. (2020); Saunders and Byrne (2020);…”
Cyberbullying Detection With Fairness Constraints (2020, methodology): “Fairness of machine learning models can be assessed by the metrics False Negative Equality Difference (FNED) and False Positive Equality Difference (FPED), as done in several studies.(6,8,9) FNED/FPED is simply the sum of deviations of group-specific false negative rates (FNRs)/false positive rates ”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “Dataset Sampling Lexical Annotation Racial Gender E E M E E M E M Founta [34] [69, 113] [113] [120] [19, 20, 44] [85, 116, 120] [44] [70]”
Language-Driven Image Style Transfer (2021, background): “Regarding guided instructions, for example, hate speech detection [91, 92, 93, 94] can help to ﬁlter out malicious texts and prevent from producing controversial results with ethics concerns.”
Benchmarking Intersectional Biases in NLP (2022, background): “The Multilingual Twitter Corpus (MTC) hate-speech dataset contains labeled Twitter messages for the task of hate speech detection (Huang et al., 2020).”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Huang et al. (2020), for example, show significant differences in social bias for multilingual hate speech systems across different languages.”
Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation (2022, background): “(6) As effects of comment authors’ demographic differences and not the annotators’ demographics as we study [20, 25, 27, 43], or creating new data sets based on political sub-communities of authors [40].”
Few-shot Learning with Multilingual Generative Language Models (2021, methodology): “We adopt datasets introduced by Huang et al. (2020) that include hate speech data from Twitter in five languages: English, Italian, Portuguese, Polish and Spanish.”
Theories of “Gender” in NLP Bias Research (2022, background): “[64] Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J.”