Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines

By M. Sabou, Kalina Bontcheva, Leon Derczynski, Arno Scharl. 2014

189 citations recorded by the source corpus.

Crowdsourcing is an emerging collaborative approach that can be used for the acquisition of annotated corpora and a wide range of other linguistic resources. Although the use of this approach is intensifying in all its key genres (paid-for crowdsourcing, games with a purpose, volunteering-based approaches), the community still lacks a set of best-practice guidelines similar to the annotation best practices for traditional, expert-based corpus acquisition. In this paper we focus on the use of crowdsourcing methods for corpus acquisition and propose a set of best practice guidelines based in our own experiences in this area and an overview of related literature. We also introduce GATE Crowd, a plugin of the GATE platform that relies on these guidelines and offers tool support for using crowdsourcing in a more principled and efficient manner.

Subfields: Communication, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Computer Science Applications

In Evaluation & Benchmarks

Citations in the corpus

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation (2022, neutral): “, through majority voting) to obtain a single “ground truth” that is released in the dataset [45].”
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (2019, background): “In a typical process, annotators are recruited and screened (Sabou et al., 2014), often resulting in a small group that creates most of the dataset examples.”
Challenges and frontiers in abusive content detection (2019, neutral): “In line with the recommendations of Sabou et al., we advocate that annotation guidelines and processes are shared where possible (Sabou, Bontcheva, Derczynski, & Scharl, 2014) and that the field also works to develop best practices.”
Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment (2018, methodology): “As the former method, majority vote of human annotators was widely used (Sabou et al., 2014; Bowman et al., 2015; Al Khatib et al., 2016) As the latter metric, many kind of inter-annotator agreement metrics based on multiple human annotation results were employed, because direct assessment of data…”
FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “For consistency, raters were asked to validate groups of related classes (e.g., sibling categories), so they could get familiar with speciﬁc sections of the ontology [66].”
Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from? (2019, neutral): “round standardization and reliability of linguistic tagging. Linguistics researchers have long developed best practices for corpus annotation [e.g. 24], including recent work about using crowdworkers [52]. Annotated corpus projects often release guidelines and reflections about their process. For ex”
Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference (2019, background): “A challenge of crowdsourcing is that workers are untrained and some might produce low quality annotations (Sabou et al., 2014).”
Introducing CAD: the Contextual Abuse Dataset (2021, background): “They are cheap and scalable but can be low quality and are often ill-suited to complicated tasks (Sabou et al., 2014).”
Analysis of named entity recognition and linking for tweets (2014, background): “One way to address this data challenge is through crowdsourcing such corpora in their entirety (Sabou et al., 2014).”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, methodology): “In case of disagreements between annotations, they are often aggregated by majority voting, averaging (Sabou et al., 2014), or adjudicating by an ‘ex-pert’ (Waseem and Hovy, 2016), to derive a singular ground truth or gold label that is later used for training supervised machine learning models.”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, background): “This model is possibly most widely applied to linguistic data (Plank et al., 2014; Sabou et al., 2014; Habernal and Gurevych, 2016).”
Freesound Datasets: A Platform for the Creation of Open Audio Datasets (2017, background): “Different kinds of crowdsourcing approaches have been explored, which mainly vary in the way users are rewarded by their contributions, including volunteering-based approaches, games with a purpose, and paid-for crowdsourcing [28].”
Data Collection for Dialogue System: A Startup Perspective (2018, background): “However, designing effective methodologies for data collection with the crowd is largely an open research question (Sabou et al., 2014).”
Directions in Abusive Language Training Data: Garbage In, Garbage Out (2020, methodology): “Crowdsourcing has also been associated with ‘citizen science’ initiatives to make academic research more accessible but this may not be fully realised in cases where annotation tasks are laborious and low-skilled [16, 91].”
Directions in abusive language training data, a systematic review: Garbage in, garbage out (2020, background): “Crowdsourcing has also been associated with ‘citizen science’ initiatives to make academic research more accessible but this may not be fully realised in cases where annotation tasks are laborious and low-skilled [50, 108].”