Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation

By Lora Aroyo, Chris Welty. 2015

455 citations recorded by the source corpus.

Big data is having a disruptive impact across the sciences. Human annotation of semantic interpretation tasks is a critical part of big data semantics, but it is based on an antiquated ideal of a single correct truth that needs to be similarly disrupted. We expose seven myths about human annotation, most of which derive from that antiquated ideal of truth, and dispell these myths with examples from our research. We propose a new theory of truth, crowd truth, that is based on the intuition that human interpretation is subjective, and that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Evaluation & Benchmarks

Citations in the corpus

LaMDA: Language Models for Dialog Applications (2022, background): “Future work will include selecting crowdworkers that mirror the system’s target users, and looking at ways to improve the quality of labels, through training and evaluation approaches that also account for systematic disagreements between crowdworkers due to social and cultural norms and values [99]”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “le 21Wemeasurecorrectnessinsteadofinter-annotatoragreement since question may have multiple correct answers. For example, We have observed a yes/no question where both YES and NO were deemed correct. Aroyo and Welty (2015) discuss the pitfalls of over-constrained annotation guidelines in depth. 22Tr”
Teaching language models to support answers with verified quotes (2022, background): “7Standard approaches to improving dataset quality can exacerbate this by collapsing a diversity of opinions to the majority vote (Aroyo and Welty, 2015)”
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, background): “As Aroyo and Welty [5] argue, the notion of “one truth” in crowdsourcing responses is a myth; disagreement between annotators, which is often viewed as problematic noise, can actually provide a valuable signal.”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “More generally, several researchers have identified a widespread failure to recognize annotation work as interpretive work, which in turn can result in a conflation of target labels in a collected dataset and real-world objects, for which there may be no single ground truth label.(55,56) As discusse”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “HCI research has long considered the ideal of crowdsourcing a single ground-truth as a “myth” that fails to account for the diversity of human thought and experience [9].”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “High-stakes AI requires specialised, subjective decision-making in defning the ground truth, and breadth and number of labels [13].”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “HCI research has long considered the ideal of crowdsourcing a single ground-truth as a “myth” that fails to account for the diversity of human thought and experience (Aroyo and Welty, 2015).”
Decontextualization: Making Sentences Stand-Alone (2021, background): “While this inherent subjectivity makes task challenging to crowdsource and evaluate, we argue this is important feature, as shown in recent literature (Aroyo and Welty, 2015; Pavlick and Kwiatkowski, 2019; Kwiatkowski et al., 2019), and propose to collect multiple references per example.”
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021, background): “…discussion on minimal standards for inter-annotator agreement to ensure data quality (Di Eugenio and Glass, 2004; Passonneau, 2004; Artstein and Poesio, 2008), recently an increasing number of works argue that disagreement is unavoidable because language is inherently ambiguous (Aroyo and…”
Implicit Bias in Crowdsourced Knowledge Graphs (2019, background): “An alternative approach to aggregating answers is to analyze information about disagreement among workers [1] or to compute confidence measures about the answer [11] by means of appropriate agreement measures [4].”
AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images (2016, background): “The most astonishing fact about crowdsourcing studies in the medical domain, however, is the conclusion that a crowd of non-professional, inexperienced users do not underperform medical experts [9], [15].”
Evaluation Gaps in Machine Learning Practice (2022, background): “An array of recent scholarship on subjectivity in ML has “embraced disagreement” through practices of explicitly modeling—in both the data model and the ML model—inter-subject variation in interpretations [7, 12, 45, 48, 55].”