UNQOVERing Stereotypical Biases via Underspecified Questions

By Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Vivek Srikumar. 2020

167 citations recorded by the source corpus.

While language embeddings have been shown to have stereotyping biases, how these biases affect downstream question answering (QA) models remains unexplored. We present UNQOVER, a general framework to probe and quantify biases through underspecified questions. We show that a naive use of model scores can lead to incorrect bias estimates due to two forms of reasoning errors: positional dependence and question independence. We design a formalism that isolates the aforementioned errors. As case studies, we use this metric to analyze four important classes of stereotypes: gender, nationality, ethnicity, and religion. We probe five transformer-based QA models trained on two QA datasets, along with their underlying language models. Our broad study reveals that (1) all these models, with and without fine-tuning, have notable stereotyping biases in these classes; (2) larger models often have higher bias; and (3) the effect of fine-tuning on bias varies strongly with the dataset and the model size.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “…and profession (Bolukbasi et al., 2016; Rudinger et al., 2018b; Zhao et al., 2018), negative sentiment about different racial and religious groups (Li et al., 2020; Nadeem et al., 2021), and persons with disabilities (Hutchinson et al., 2020), as well as other social biases (Caliskan et al.,…”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “To reproduce the ethnicity bias results of Li et al. (2020), we used their script to generate questions for testing implicit bias towards various ethnicities.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Popular evaluations for biases use association tests [46] or examine stereotypes [157, 156, 138], correlations and co-occurrences [272], and sentiment analysis [66].”
BBQ: A hand-built bias benchmark for question answering (2021, background): “suring social bias in QA models, UnQover (Li et al., 2020).”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021, methodology): “We use the comparative measure of bias score C(x1, x2, q, τ) (Li et al., 2020, Section 4.2; Eq.”
Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021, background): “Although this is still an active area of research, for now, these are promising proxies that many work in ethical NLP also have used to measure bias (e.g. Sheng et al. (2019); Li et al. (2020); Brown et al. (2020); Sheng et al. (2020); Dhamala et al. (2021)).”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “For several tasks, metrics like this already exist, at least for gender in English, in the form of auxiliary test sets meant to be combined with a pre-existing training set (Rudinger et al., 2018; Webster et al., 2018; Kiritchenko and Mohammad, 2018; Li et al., 2020).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, methodology): “Similarly to BBQ, UnQover [67] studies language model biases by asking ambiguous questions.”
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, background): “In response to such concerns, various benchmarks have been proposed to quantify the amount of social biases in models (Rudinger et al., 2018; Sheng et al., 2019; Li et al., 2020).”
Analyzing Stereotypes in Generative Text Inference Tasks (2021, background): “Li et al. (2020) probes transformer-based question answering models on stereotypes towards gender, nationality, religion, ethnicity domains.”
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021, background): “Another direction for future work is accuracy under counterfactual perturbation: swapping real-world entities (in contrast with nonce entities in Li et al. (2020)) with different demographic values.”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “On the other hand, its reliance on their parameters to encode a wealth of world knowledge learned has several limitations: they often struggle to memorize less popular knowledge and can result in biased predictions (Li et al., 2020; Parrish et al., 2022) and their knowledge gets obsolete quickly.”
On Measures of Biases and Harms in NLP (2021, methodology): “Li et al. (2020) use sentence templates containing limited direct demographic information (e.g., names) and under-specified questions containing no related demographic information to measure biases exhibited by QA systems.”
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023, background): “In fact, keeping the pre-training data not available for LLM consumers may derive undesired influences on downstream tasks (Li et al., 2020; Gehman et al., 2020; Groen-wold et al., 2020).”
Bias and Fairness in Large Language Models: A Survey (2023, background): “While BBQ provides a correct answer, all answers in UnQover indicate a stereotyping bias, because each answer should be equally likely under an unbiased model.”
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models (2023, background): “…example, some work concep-tualises stereotypes as commonly held beliefs about particular demographic groups (and anti-stereotypes as their inverse) (Li et al., 2020), while others conceptualise stereotypes as negative beliefs (Zhou et al., 2022; Dinan et al., 2022), possibly conflating negative…”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Mitigating Model Bias: It has already been shown that gender bias exists in NMT models when trained on high-resource languages [118, 174].”
Large Language Model Alignment: A Survey (2023, background): “UnQover utilizes unspecified questions, which couldn’t be answered simlpy according to the given context.”
Perturbation Augmentation for Fairer NLP (2022, background): “In line with prior work on predictive biases in ML systems (Li et al., 2020; Parrish et al., 2021), we find that classification models are generally not very robust to demographic perturbation.”