Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers

By Andrew Silva, Pradyumna Tambwekar, Matthew Craig Gombolay. 2021

66 citations recorded by the source corpus.

The ease of access to pre-trained transformers has enabled developers to leverage large-scale language models to build exciting applications for their users. While such pre-trained models offer convenient starting points for researchers and developers, there is little consideration for the societal biases captured within these model risking perpetuation of racial, gender, and other harmful biases when these models are deployed at scale. In this paper, we investigate gender and racial bias across ubiquitous pre-trained language models, including GPT-2, XLNet, BERT, RoBERTa, ALBERT and DistilBERT. We evaluate bias within pre-trained transformers using three metrics: WEAT, sequence likelihood, and pronoun ranking. We conclude with an experiment demonstrating the ineffectiveness of word-embedding techniques, such as WEAT, signaling the need for more robust bias testing in transformers.

Subfields: Artificial Intelligence, General Social Sciences, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nanyun Peng
2. Emily Sheng
3. Kai-Wei Chang
4. Premkumar Natarajan
5. Isabelle Augenstein
6. Karolina Stańczak
7. Henrik Björklund
8. Hannah Devinney

In General Social Sciences

1. Isabelle Augenstein
2. Karolina Stańczak
3. Matthew Craig Gombolay
4. Premkumar Natarajan
5. Andrew Silva
6. Emily Sheng
7. Franck Dernoncourt
8. Kai-Wei Chang

In Safety, Robustness & Interpretability

1. Emily Sheng
2. Nanyun Peng
3. Kai-Wei Chang
4. Premkumar Natarajan
5. Isabelle Augenstein
6. Karolina Stańczak
7. Henrik Björklund
8. Hannah Devinney

Citations in the corpus

• Societal Biases in Language Generation: Progress and Challenges (2021, background): “Silva et al. (2021) observe amplified biases in distilled versus original models.”
• Cross-Loss Influence Functions to Explain Deep Network Representations (2020, background): “Bolukbasi et al. (2016) were among the first to discover the inherent biases in word embeddings, a finding that has been replicated in transformers (Basta et al., 2019; Kurita et al., 2019; Silva et al., 2021) and more (Costa-jussà et al., 2019; Swinger et al., 2019; Nissim et al., 2020).”
• A Survey on Gender Bias in Natural Language Processing (2021, methodology): “However, Sun et al. [2019] note that it is unclear if gender-neutralised word embeddings are applicable to languages with grammatical genders.”
• Theories of “Gender” in NLP Bias Research (2022, background): “03133 [138] Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay.”
• Bias and Fairness in Large Language Models: A Survey (2023, background): “During distilation, the student model may inherit or even amplify biases from the teacher (Ahn et al., 2022; Silva et al., 2021).”