Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard

By Lora Aroyo, Chris Welty. 2013

132 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation (2015, methodology): “The three kinds of scores hold up well in our experiments for building a medical relation extraction gold standard (Aroyo and Welty 2013b).”
On Releasing Annotator-Level Labels and Information in Datasets (2021, background): “Enforcing such a single ground truth in such tasks will sacrifice valuable nuances about the task that are embedded in annotators’ assessments of the stimuli, especially their disagreements (Aroyo and Welty, 2013).”
Why Gender and Age Prediction from Tweets is Hard: Lessons from a Crowdsourcing Experiment (2014, background): “While variance between annotators has traditionally been treated as noise, more recently variation is being treated as a signal rather than noise (Aroyo and Welty, 2013).”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, background): “Aroyo and Welty (2013) also studied inter-annotator disagreement in association with features of the input, showing that it reﬂects semantic ambiguity of the training instances.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “However, this assumption has been shown to be inaccurate for some tasks due to a low inter-annotator agreement (Aroyo and Welty, 2013; Pavlick and Kwiatkowski, 2019; Elazar et al., 2020; Nie et al., 2020), and the extent to which it is true likely varies by task, instance, and annotator.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “However, this assumption has been shown to be inaccurate (among others) for relation extraction [8], natural language inference [96, 88], and complement coercion [35], and the extent to which it is true likely varies by task, instance, and annotator.”