“You Are Grounded!”: Latent Name Artifacts in Pre-trained Language Models

By Vered Shwartz, Rachel Rudinger, Oyvind Tafjord. 2020

55 citations recorded by the source corpus.

Pre-trained language models (LMs) may perpetuate biases originating in their training corpus to downstream models. We focus on artifacts associated with the representation of given names (e.g., Donald), which, depending on the corpus, may be associated with specific entities, as indicated by next token prediction (e.g., Trump). While helpful in some contexts, grounding happens also in under-specified or inappropriate contexts. For example, endings generated for `Donald is a' substantially differ from those of other names, and often have more-than-average negative sentiment. We demonstrate the potential effect on downstream tasks with reading comprehension probes where name perturbation changes the model answers. As a silver lining, our experiments suggest that additional pre-training on different corpora may mitigate this bias.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, background): “To gain more insight into biases from different decoding techniques, we examine autocomplete generations from GPT (110M params), GPT-2 (small, 117M params), and XLNet (base, 110M params), using the decoding techniques described in Section 4.3 through the Transformers 19 library.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, methodology): “Names have been used as demographic proxies in both social computing and NLP work, and often appear in close proximity to other demographic markers in text corpora; LMs might thus memorize incidental associations between names and demographic criteria (Prabhakaran et al., 2019; Shwartz et al., 2020)”
Towards Controllable Biases in Language Generation (2020, background): “Prabhakaran et al. (2019) show that NLP models are susceptible to learn different incidental associations with different names, and Shwartz et al. (2020) further analyze name biases in language models.”
Entity-Based Knowledge Conflicts in Question Answering (2021, methodology): “Using popularity substitu-tion we examine if models are biased towards predicting more popular answers (Shwartz et al., 2020; Chen et al., 2021).”
Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media (2021, result): “This result suggest that the pre-trained DistilBERT model itself is likely responsible for the latent bias, mirroring multiple re-sults showing biases in these models (Kurita et al., 2019b; Shwartz et al., 2020; Nadeem et al., 2020).”
On Measures of Biases and Harms in NLP (2021, background): “There are also other bias measures for language models that rely on sentiment (Groenwold et al., 2020; Shwartz et al., 2020).”
Do Neural Language Models Overcome Reporting Bias? (2020, methodology): “The results in this paper are in line with prior work that showed that LMs amplify social bias (May et al., 2019; Sheng et al., 2019) and knowledge about named entities that are prominent in the corpus (Shwartz et al., 2020a).”
What do Bias Measures Measure? (2021, background): “There are also other bias measures for language models that rely on sentiment (Groenwold et al., 2020; Shwartz et al., 2020 2020) evaluate for similar biases in language models towards people who produce the text (Sheng et al., 2021).”
Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data (2020, background): “For a discussion of related problem see (Shwartz et al., 2020).”
Gender and Representation Bias in GPT-3 Generated Stories (2021, background): “An ongoing line of research examines the nature and effects of these biases in natural language generation (Sheng et al., 2020; Wallace et al., 2019; Shwartz et al., 2020).”
Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution (2022, background): “Due to biases in training data, pre-trained language models may not treat all names equally on downstream tasks (Shwartz et al., 2020).”