Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

By Mor Geva, Yoav Goldberg, Jonathan Berant. 2019

354 citations recorded by the source corpus.

Crowdsourcing has been the prevalent paradigm for creating natural language understanding datasets in recent years. A common crowdsourcing practice is to recruit a small number of high-quality workers, and have them massively generate examples. Having only a few workers generate the majority of examples raises concerns about data diversity, especially when workers freely generate sentences. In this paper, we perform a series of experiments showing these concerns are evident in three recent NLP datasets. We show that model performance improves when training with annotator identifiers as features, and that models are able to recognize the most productive annotators. Moreover, we show that often models do not generalize well to examples from annotators that did not contribute to the training set. Our findings suggest that annotator bias should be monitored during dataset creation, and that test set annotators should be disjoint from training set annotators.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “…that state-of-the-art models learn to exploit spurious statistical patterns in datasets (Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Glockner et al., 2018; Geva et al., 2019; McCoy et al., 2019), instead of learning meaning in the ﬂexible and generalizable way that humans do.”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, methodology): “(Gururangan et al., 2018; Geva et al., 2019).”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “by learning simple decision rules rather than encoding a more generalisable understanding of the task (e.g. Niven and Kao, 2019; Geva et al., 2019; Shah et al., 2020).”
Shortcut learning in deep neural networks (2020, background): “Shortcut learning starts from various dataset biases such as annotation artefacts [55, 56, 57, 58].”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Existing datasets are often biased due to their curation process (Dixon et al., 2018; Wiegand et al., 2019; Geva et al., 2019; Sap et al., 2021; Kryscinski et al., 2020) and can have unreliable annotations (Ross et al., 2017; Field and Tsvetkov, 2020b; Pagnoni et al., 2021).”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “In order to validate that our system is not overfitting to a single group of annotators (Geva et al., 2019), we collect an additional set of annotations for the test set of SNLI from a group of annotators who did not participate in any of our previous annotation tasks (“Test_Crowd2”).”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “Second, these annotations can help combat biases that are often introduced while collecting data (Gururangan et al., 2018; Geva et al., 2019).”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “…approach is known to result in machine-exploitable annotator artefacts (Jia and Liang, 2017; Schwartz et al., 2017; Gururangan et al., 2018; Geva et al., 2019), leading to poor out-of-distribution generalisation (Chen et al., 2016; Weissenborn et al., 2017; Yogatama et al., 2019; McCoy et…”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “Data Generation Although manual creation of datasets is expensive and introduces annotation artifacts (Schwartz et al., 2017; Gururangan et al., 2018; Agrawal et al., 2018; Tsuchiya, 2018; Geva et al., 2019; Bras et al., 2020), crowdsourcing remains the most popular method for curating goaloriented ”
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021, background): “This makes the test sets more challenging and minimizes the risk of annotator bias given our relatively small pool of annotators (Geva et al., 2019).”
Are We Done with MMLU? (2024, neutral): “…[28, 29, 30]; automated labelling processes such as crowdsourcing [31] — where annotators may be optimizing for their own incentives; human errors [32] — e.g., lack of expertise on a given topic; or programmatic weak supervision [33, 34, 7], and personal [35] or group-level [36] annotator biases.”
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020, background): “, 2019), labeling errors, annotator biases (Geva et al., 2019), worst-case security issues (Wallace et al.”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “We argure that the underlying reasons are two-fold: 1) The creation of distractor choices has annotator bias (Geva et al., 2019) which can be easily detected by NLU models.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “In particular, one common source is annotator bias from data collection processes (Geva et al., 2019).”
Reasoning Over Paragraph Effects in Situations (2019, neutral): “In initial experiments, we found splitting the dataset based on the situations resulted in high scores due to annotator bias from proliﬁc workers generating many examples (Geva et al., 2019).”
Adversarial Filters of Dataset Biases (2020, background): “, 2018) or during human annotation (Rudinger et al., 2017; Gururangan et al., 2018; Poliak et al., 2018; Tsuchiya, 2018; Geva et al., 2019).”