Effective Crowd Annotation for Relation Extraction

By Angli Liu, Stephen Soderland, Jonathan Bragg, C. H. Lin, Xiao Ling, Daniel S. Weld. 2016

83 citations recorded by the source corpus.

Can crowdsourced annotation of training data boost performance for relation extraction over methods based solely on distant supervision? While crowdsourcing has been shown effective for many NLP tasks, previous researchers found only minimal improvement when applying the method to relation extraction. This paper demonstrates that a much larger boost is possible, e.g., raising F1 from 0.40 to 0.60. Furthermore, the gains are due to a simple, generalizable technique, Gated Instruction , which combines an interactive tutorial, feedback to correct errors during training, and improved screening.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Natural Language Processing

Citations in the corpus

Zero-Shot Relation Extraction via Reading Comprehension (2017, methodology): “When the type of facts (relations) are predeﬁned, one can use crowdsourcing (Liu et al., 2016) or distant supervision (Hoffmann et al., 2011) to collect examples and train an extraction model for each relation type.”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “Similarly, Liu et al. (2016b) ﬁnd correlations less than 0 .”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, neutral): “Human evaluations, such as those given on Amazon Mechanical Turk [49, 14], remain ad-hoc because “results change drastically” [52] based on details of the task design [36, 34, 27].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, unknown): “; and they were shown dataset-specific examples. 2) To familiarize them with the task and verify their understanding, a screening phase required the participant to correctly label four of six reviews [44]. Only participants who passed the gating test were allowed to proceed to the main task. 3) The ”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “● Training protocols can be used to prepare workers for the job, which can have a positive impact on their resulting performance [17].”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality (2016, background): “Once we have identified these workers, we can back off the goldstandard (attention check) questions to ensure good quality work, since work quality is unvarying [37].”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “They used two relation datasets: NYT dataset [111] and UW dataset [80].”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Accuracy (2016, background): “Once we have identified these workers, we can back off the goldstandard (attention check) questions to ensure good quality work, since work quality is unvarying [36].”
Is GPT-3 a Good Data Annotator? (2022, methodology): “Data annotation is the process of labeling data to help machine learning models better understand the purpose of any given task (Liu et al., 2016; Rogers, 2021), recognize patterns and make predictions (Tan et al.”