On Measuring Social Biases in Sentence Encoders

By Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, Rachel Rudinger. 2019

730 citations recorded by the source corpus.

The Word Embedding Association Test shows that GloVe and word2vec word embeddings exhibit human-like implicit biases based on gender, race, and other social constructs (Caliskan et al., 2017). Meanwhile, research on learning reusable text representations has begun to explore sentence-level texts, with some sentence encoders seeing enthusiastic adoption. Accordingly, we extend the Word Embedding Association Test to measure bias in sentence encoders. We then test several sentence encoders, including state-of-the-art methods such as ELMo and BERT, for the social biases studied in prior work and two important biases that are difficult or impossible to test at the word level. We observe mixed results including suspicious patterns of sensitivity that suggest the test’s assumptions may not hold in general. We conclude by proposing directions for future work on measuring bias in sentence encoders.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “A large body of work analyzing “bias” in natural language processing (NLP) systems has emerged in recent years, including work on “bias” in embedding spaces (e.g., Bolukbasi et al., 2016a; Caliskan et al., 2017; Gonen and Goldberg, 2019; May et al., 2019) as well as work on “bias” in systems develop”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, background): “3.2 Bias in pretrained language models May et al. (2019) extend WEAT to sentence encoders, calling it the Sentence Encoder Association Test (SEAT).”
Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, methodology): “2019), the analogy task is not used anymore to either evaluate their soundness, or to detect bias (Zhao et al. 2019; Basta, Costa-jussà, and Casas 2019; May et al. 2019).”
LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “, 2021) and proposed metrics to assess the degree to which language models encode harmful stereotypes intrinsically in their representations (May et al., 2019; Webster et al., 2021).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “A wealth of work has shown that toxicity and social biases in training data are acquired by large pretrained sentence encoders (e.g., gender bias in BERT; May et al., 2019; Zhao et al., 2019; Basta et al., 2019; Kurita et al., 2019).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, background): “…(Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; Barocas et al., 2017; Garg et al., 2019; Dixon et al., 2018; Noble, 2018; Manzini et al., 2019; Sap et al., 2019; May et al., 2019; Speer, 2017), bias with respect to different disability groups has been relatively under-explored.”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, methodology): “May et al. (2019) extend Caliskan et al.’s analysis to sentence-level evaluation with the SEAT test set.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Moreover, several other metrics have been developed for measuring societal biases in contextualized word representation (Kurita et al., 2019; May et al., 2019; Guo and Caliskan, 2021) which are extensions of Word Embedding Association Test (WEAT) (Caliskan et al., 2017).”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “WEAT/SEAT, which measure bias in static word embedding using cosine similarity of specific target words (Caliskan et al., 2017; May et al., 2019).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “Such prompts are generally derived either from crowdsourcing (Nadeem et al., 2021; Nangia et al., 2021) or from slotting a set of terms into templates (Kurita et al., 2019; May et al., 2019; Sheng et al., 2019; Webster et al., 2020).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “, 2020)? Do our models’ gender biases intersect with racialized biases (Davis, 1981; Crenshaw, 1989; May et al., 2019; Tan and Celis, 2019)?”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “Prior work has explored the extent to which language models encode harmful gender and racial biases that parallel humans through the Word Embedding Association Test (WEAT) (Caliskan et al., 2017), the Sentence Encoder Association Test (SEAT) (May et al., 2019) and the GroundedWEAT/Grounded-SEAT (Ros”
Towards Understanding and Mitigating Social Biases in Language Models (2021, methodology): “This further generalizes WEAT (Caliskan et al., 2017) or SEAT (May et al., 2019) tests by comparing across all tokens while at the same time weighting more likely tokens higher in bias computation, instead of only considering a predeﬁned set of bias attributes (e.g., gendered terms and occupations).”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, methodology): “Choosing a suitable metric with which to analyse bias is a key challenge (May et al., 2019); whilst a positive result with respect to a suitable metric does reveal the existence of bias, a negative result does not mean a model is completely bias-free.”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “In [36], their Sentence Encoder Association Test (SEAT) based on Word Embedding Association Test (WEAT [24]) is introduced in the context of sentence encoders.”
Measuring Bias in Contextualized Word Representations (2019, methodology): “Recently, May et al. (2019) showed that traditional cosine-based methods for exposing bias in sentence embeddings fail to produce consistent results for embeddings generated using contextual methods.”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “…this limitations, researchers passed single sentences through BERT and then derive a ﬁxed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: May et al. (2019); Zhang et al. (2019); Qiao et al. (2019)).”