Perturbation Sensitivity Analysis to Detect Unintended Model Biases

By Vinodkumar Prabhakaran, Ben Hutchinson, Margaret Mitchell. 2019

132 citations recorded by the source corpus.

Data-driven statistical Natural Language Processing (NLP) techniques leverage large amounts of language data to build models that can understand language. However, most language data reflect the public discourse at the time the data was produced, and hence NLP models are susceptible to learning incidental associations around named referents at a particular point in time, in addition to general linguistic meaning. An NLP system designed to model notions such as sentiment and toxicity should ideally produce scores that are independent of the identity of such entities mentioned in text and their social associations. For example, in a general purpose sentiment analysis system, a phrase such as I hate Katy Perry should be interpreted as having the same sentiment as I hate Taylor Swift. Based on this idea, we propose a generic evaluation framework, Perturbation Sensitivity Analysis, which detects unintended model biases related to named entities, and requires no new annotations or corpora. We demonstrate the utility of this analysis by employing it on two different NLP models — a sentiment model and a toxicity model — applied on online comments in English language from four different genres.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…Font and Costa-jussà (2019); Garimella et al. (2019); Loukina et al. (2019); Mayfield et al. (2019); Mehrabi et al. (2019); Nozza et al. (2019); Prabhakaran et al. (2019); Romanov et al. (2019); Ruane et al. (2019); Sap et al. (2019); Sheng et al. (2019); Sun et al. (2019); Sweeney and…”
LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “perturbation testing [156], data augmentation [220], adversarial testing [117], and adherence to terminology guidelines for oppressed groups, such as SIGACCESS.”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, methodology): “Following (Garg et al., 2019; Prabhakaran et al., 2019), we use the notion of perturbation, whereby the phrases for referring to people with disabilities, described above, are all inserted into the same slots in sentence templates.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, methodology): “Names have been used as demographic proxies in both social computing and NLP work, and often appear in close proximity to other demographic markers in text corpora; LMs might thus memorize incidental associations between names and demographic criteria (Prabhakaran et al., 2019; Shwartz et al., 2020)”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “turbation sensitivity analysis (Prabhakaran et al., 2019) that reveals biases by counterfactual replacement of terms of same semantic category in natural sentences.”
Towards Controllable Biases in Language Generation (2020, background): “Prabhakaran et al. (2019) show that NLP models are susceptible to learn different incidental associations with different names, and Shwartz et al. (2020) further analyze name biases in language models.”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, methodology): “Out of the 26 metrics we ﬁnd, only four do not ﬁt within our framework: BPSN and BNSP (Borkan et al., 2019), the metric (De-Arteaga et al., 2019) and Perturbation Label Distance (Prabhakaran et al., 2019).”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “For instance, toxicity prediction models were shown to have biases towards mentions of certain identity terms (Dixon et al., 2018), speciﬁc named entities (Prabhakaran et al., 2019), and disabilities (Hutchinson et al., 2020).”
Large language models encode clinical knowledge (2022, background): “For instance, depending on the context, it may be relevant to assess sensitivity of model outputs to perturbations of demographic identifiers in prompts designed deliberately such that the result should not change under the perturbation [23, 68, 98].”
“You Are Grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020, background): “Prabhakaran et al. (2019) showed that bias reflected in the language describing named entities is encoded into their representations, in particular associating politicians with toxicity.”
On Measures of Biases and Harms in NLP (2021, background): “However, this task has been demonstrated to be stereotypically influenced by demographic characteristics such as race and gender (Kiritchenko and Mohammad, 2018), age (Díaz et al., 2018) and names of individuals (Prabhakaran et al., 2019).”
A Study of Implicit Bias in Pretrained Language Models against People with Disabilities (2022, methodology): “We then generate adjectives <adj> for each group based on the perturbation technique introduced by Prabhakaran et al. (2019).”
What do Bias Measures Measure? (2021, background): “However, this task has been demonstrated to be stereotypically inﬂuenced by demographic characteristics such as race and gender (Kiritchenko and Mohammad, 2018), age (Díaz et al., 2018) and names of individuals (Prabhakaran et al., 2019).”
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021, methodology): “However, PSA (and similar approaches) aggregate the sensitivity to terms by calculating the mean difference in prediction scores across all sentences.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, background): “higher levels of toxicity with sentences containing identity markers for marginalized groups or even specific names [61, 103].”
Evaluation Gaps in Machine Learning Practice (2022, methodology): “Our framework for analyzing the gaps builds upon and complements other streams of work on ML evaluation practices, including addressing distribution shifts between development data and application data [34, 94, 160], and robustness to perturbations in test items [118, 132, 173 ?”