Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media

By Sayan Ghosh, Dylan Baker, David Jurgens, Vinodkumar Prabhakaran. 2021

47 citations recorded by the source corpus.

Online social media platforms increasingly rely on Natural Language Processing (NLP) techniques to detect abusive content at scale in order to mitigate the harms it causes to their users. However, these techniques suffer from various sampling and association biases present in training data, often resulting in sub-par performance on content relevant to marginalized groups, potentially furthering disproportionate harms towards them. Studies on such biases so far have focused on only a handful of axes of disparities and subgroups that have annotations/lexicons available. Consequently, biases concerning non-Western contexts are largely ignored in the literature. In this paper, we introduce a weakly supervised method to robustly detect lexical biases in broader geo-cultural contexts. Through a case study on a publicly available toxicity detection model, we demonstrate that our method identifies salient groups of cross-geographic errors, and, in a follow up, demonstrate that these groupings reflect human judgments of offensive and inoffensive language in those geographic contexts. We also conduct analysis of a model trained on a dataset with ground truth labels to better understand these biases, and present preliminary mitigation experiments.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Re-contextualizing Fairness in NLP: The Case of India (2022, background): “Research on undesirable biases has been a growing priority in NLP (Caliskan et al., 2017; Blodgett et al., 2020; Sheng et al., 2021; Ghosh et al., 2021).”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, background): “Since many crowdsourced annotator poolsare sociodemograph-ically skewed, there are implications for which populations and cultural values are represented in datasets and models [20] as well as which populations face the challenges of crowdwork [22, 30].”
Cultural Incongruencies in Artificial Intelligence (2022, background): “For instance, content moderation systems meant to detect offensiveness and misinformation may miss culture-speciﬁc offensive terms and interpretations allowing toxic or violent speech to propagate for some cultural settings [21, 14].”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “As extensive research has shown that machine learning models exhibit societal and political biases (Zhao et al., 2018; Blodgett et al., 2020b; Bender et al., 2021; Ghosh et al., 2021; Shaikh et al., 2022; Li et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021; Jin et al., 2021), there has ”
Socially Aware Bias Measurements for Hindi Language Representations (2021, unknown): “Additionally, Ghosh et al. (2021) and Sambasivan et al. (2020) explore biases and algorithmic fairness concerning non-western contexts in machine learning and Pujari et al. (2019) focus on fairness in language technologies for Indic society by investigating binary gender associated bias in Hindi…”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “[37] then proposed a two-step weakly-supervised method to detect lexical bias for cross-geocultural harmful content.”