Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification

By Daniel Borkan, Lucas Dixon, Jeffrey Scott Sorensen, Nithum Thain, Lucy Vasserman. 2019

607 citations recorded by the source corpus.

Unintended bias in Machine Learning can manifest as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large. In this paper, we introduce a suite of threshold-agnostic metrics that provide a nuanced view of this unintended bias, by considering the various ways that a classifier’s score distribution can vary across designated groups. We also introduce a large new test set of online comments with crowd-sourced annotations for identity references. We use this to show how our metrics can be used to find new and potentially subtle unintended bias in existing public models.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

In Evaluation & Benchmarks

Citations in the corpus

Challenges in Detoxifying Language Models (2021, methodology): “…API for both filtering and evaluation, we filter with a separate BERT-based toxicity classifier (Devlin et al. (2019), denoted as BERT in this work), which is finetuned for 1 epoch with a learning rate of 2×10−5 on the CIVILCOMMENTS dataset (Borkan et al., 2019), using 16 Google Cloud TPUv3 cores.”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, methodology): “We use the dataset of online comments released as part of the Jigsaw Unintended Bias in Toxicity Classification challenge (Borkan et al., 2019; Jigsaw, 2019), where a subset of 405K comments are labelled for mentions of disabilities, grouped into four types: physical disability, intellectual or…”
Pretraining Language Models with Human Preferences (2023, methodology): “(2021), we group these harms under the name of “toxicity,” understood as “a rude, disrespectful, or unreasonable comment that is somewhat likely to make you leave a discussion or give up on sharing your perspective” (Borkan et al., 2019).”
GeDi: Generative Discriminator Guided Sequence Generation (2020, methodology): “Related to the problem of addressing toxicity in generation is toxicity detection, which can be performed using the Perspective API or using a classifier trained on a labelled toxicity dataset such as the Jigsaw Toxic Comment Classification Dataset (Borkan et al., 2019).”
UL2: Unifying Language Learning Paradigms (2022, background): “…2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., 2019), VitaminC (Schuster et al., 2021a), Civil Comments and Wikipedia Toxicity detection datasets (Borkan et al., 2019).”
Perturbation Sensitivity Analysis to Detect Unintended Model Biases (2019, background): “Hence, fairness research in NLP has mostly focused on mentions of social identities (Dixon et al., 2018; Borkan et al., 2019; Garg et al., 2019), or on how social stereotypes impact semantic interpretation (Webster et al., 2018), and often rely heavily on annotated corpora.”
HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection (2020, methodology): “The models may classify comments that refer to certain commonly-attacked identities (e.g., gay, black, muslim) as toxic without the comment having any intention of being toxic (Dixon et al. 2018; Borkan et al. 2019).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Data To demonstrate C HARFORMER ’s ability to perform well on such data, we evaluate on toxicity detection using the Civil Comments (Borkan et al., 2019) and the Wikipedia Comments (Wulczyn”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, methodology): “[19] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “As Borkan et al. (2019) note, measuring subgroup AUCs can be misleading in this context, because it assumes that the classifier can set separate thresholds for different subgroups.”
Cyberbullying Detection With Fairness Constraints (2020, methodology): “We demonstrate our method on the Perspective API’s Jigsaw dataset of 403957 comments with toxicity and identity annotations.(11) We select male and female identities as groups of interest.”
Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation (2022, methodology): “This work has the following three primary contributions: (1) We explore the impact on toxicity, insult, threat, identity attack, and profanity ratings of online conversations as perceived by two groups of annotators (referred to as specialized rater pools): African American and LGBTQ American raters”
Context Sensitivity Estimation in Toxicity Detection (2021, methodology): “These posts are called toxic (Borkan et al., 2019) or abusive (Thylstrup and Waseem, 2020), and systems detecting them (Waseem and Hovy, 2016; Pavlopoulos et al., 2017b; Badjatiya et al., 2017) are called toxicity (or abusive language) detection systems.”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “Each of these comments was annotated by crowd raters (at least 3 each) for toxicity and toxicity sub-types (Borkan et al., 2019).”