[RE] Bad Seeds: Evaluating Lexical Methods for Bias Measurement

2022

Citations in the corpus

• “I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, neutral): “Finally, the process of assembling word lists itself can be tricky, as seed lexica often have several practical (Antoniak and Mimno, 2021) and conceptual (Dinan et al.”
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “…in some cases — for example, when counterfactuals can be obtained by making local substitutions of closed-class words like pronouns — but they cannot guarantee fluency or coverage of all labels and covariates of interest (Antoniak and Mimno, 2021), and are difficult to generalize across languages.”
• Evaluating Large Language Models (2022, background): “These can be general decisions involving prompting or decoding hyperparameters, or they can be bias-specific considerations such as the choice of words used to identify groups [2] or the specific mathematical form of the metrics you compute [16].”
• The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, background): “…et al., 2021; Goldfarb-Tarrant et al., 2023), not correlated in their findings of bias across intrinsic versus extrinsic techniques (GoldfarbTarrant et al., 2021; Cao et al., 2022), and susceptible to adversarial perturbations (Zhang et al., 2021) and seed word selection (Antoniak and Mimno, 2021).”
• Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models (2021, background): “Recent works investigating bias in language models have found issues with inconsistencies be-tween seed words (Antoniak and Mimno, 2021), unvoiced assumptions and data quality issues in StereoSet and CrowS-Pairs templates (Blodgett et al., 2021), and issues with semantically bleached templates (Tan…”
• Socially Aware Bias Measurements for Hindi Language Representations (2021, background): “Furthermore, we acknowledge that our analysis of gender associated biases is limited to binary gender and our intrinsic evaluations require discrete categorizations (Dev et al., 2021b; Antoniak and Mimno, 2021).”