Controlled Crowdsourcing for High-Quality QA-SRL Annotation

By Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke S. Zettlemoyer, Ido Dagan. 2019

56 citations recorded by the source corpus.

Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen. Recently, a large-scale crowdsourced QA-SRL corpus and a trained parser were released. Trying to replicate the QA-SRL annotation for new texts, we found that the resulting annotations were lacking in quality, particularly in coverage, making them insufficient for further research and evaluation. In this paper, we present an improved crowdsourcing protocol for complex semantic annotation, involving worker selection and training, and a data consolidation phase. Applying this protocol to QA-SRL yielded high-quality annotation with drastically higher coverage, producing a new gold evaluation dataset. We believe that our annotation protocol and gold standard will facilitate future replicable research of natural semantic annotations.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Information Extraction

Citations in the corpus

GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “This may be done by filtering and training crowdsourced workers, as in “controlled crowdsourcing” (Roit et al., 2020), or using qualified professional freelancers.”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, methodology): “Roit et al. (2020) collect a challenging QA-SRL dataset by carefully hiring and training crowdworkers, with a strict qualiﬁcation followed by 2 hours of training that includes giving workers extensive feedback.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “Prior work outside of EXNLP has noted the difficulty of scaling annotation to crowdworkers for complex linguistic tasks [106, 35, 99, 85].”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, background): “Another commonly used strategy when crowd-sourcing NLP datasets is to only qualify workers who pass an initial quiz or perform well in prelim-inary crowdsourcing batches (Wang et al., 2013; Cotterell and Callison-Burch, 2014; Ning et al., 2020; Shapira et al., 2020; Roit et al., 2020).”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “The difﬁculty of scaling data annotation from expert annotators to lay crowd-workers has been reported for tasks such as semantic role labelling (Roit et al., 2020), complement-coercion (Elazar et al., 2020), and discourse relation annotation (Pyatkin et al., 2020).”