Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria

By Pei-Yun Hsueh, Prem Melville, Vikas Sindhwani. HLT-NAACL 2009, 2009

339 citations recorded by the source corpus.

Annotation acquisition is an essential step in training supervised classifiers. However, manual annotation is often time-consuming and expensive. The possibility of recruiting annotators through Internet services (e.g., Amazon Mechanic Turk) is an appealing option that allows multiple labeling tasks to be outsourced in bulk, typically with low overall costs and fast completion rates. In this paper, we consider the difficult problem of classifying sentiment in political blog snippets. Annotation data from both expert annotators in a research lab and non-expert annotators recruited from the Internet are examined. Three selection criteria are identified to select high-quality annotations: noise level, sentiment ambiguity, and lexical uncertainty. Analysis confirm the utility of these criteria on improving data quality. We conduct an empirical study to examine the effect of noisy annotations on the performance of sentiment classification models, and evaluate the utility of annotation selection on classification accuracy and efficiency.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems

View Paper: https://www.semanticscholar.org/paper/4a749f461a4194f1d42d06bdd3071dd19b03ac26

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Information Systems

Citations in the corpus

Do Human Rationales Improve Machine Explanations? (2019, methodology): “In an effort to receive quality results from the crowd, we employ two strategies from crowd-sourcing research: gold standard questions and majority voting (Hsueh et al., 2009; Eickhoff and de Vries, 2013).”
A Practical Guide to Sentiment Annotation: Challenges and Solutions (2016, background): “Proceedings of NAACL-HLT 2016, pages 174–179, San Diego, California, June 12-17, 2016. c©2016 Association for Computational Linguistics”
Grammatical structures for word-level sentiment detection (2012, methodology): “Our process for obtaining gold standard data involves multiple levels of human annotation including on crowdsourcing platforms Hsueh et al. (2009).”
VizWiz::LocateIt - enabling blind people to locate objects in their environment (2010, methodology): “Mechanical Turk has been used for a wide variety of tasks, including gathering data for user studies [31], labeling image data sets used in Computer Vision research [38], and determining political sentiments in blog snippets [28].”
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021, background): “Hsueh et al. (2009), for example, identify a set of criteria to select informative yet unambiguous examples for predictive modeling in a sentiment classification task.”
The GUM corpus: creating multilayer resources in the classroom (2016, background): “While techniques such as crowd-sourcing may be used for other, potentially more intuitively resolvable tasks such as sentiment analysis (Hsueh et al. 2009), textual entailment or word sense disambiguation (Snow et al. 2008), creating, or even just correcting parses has not been treated using crowd…”
Evaluation on crowdsourcing research: Current status and future direction (2012, background): “…organizations or institutions, such as public libraries, R & D centers, government (Shah et al. 2009), etc., where mass participation (Holley 2009), scientific collaboration (Heer and Bostock 2010; Hsueh et al. 2009; Kittur et al. 2008), or citizen science (Hudson-Smith et al. 2009) take place.”
VizWiz: nearly real-time answers to visual questions (2010, background): “Mechanical Turk makes outsourcing small paid jobs practical [1] and has been used for a wide variety of purposes, including large user studies [15], labeling image data sets [30], and determining political sentiments in blog snippets [10].”
VizWiz: nearly real-time answers to visual questions (2010, background): “Mechanical Turk makes outsourcing small paid jobs practical [1] and has been used for a wide variety of purposes, including large user studies [15], labeling image data sets [30], and determining political sentiments in blog snippets [ 10 ].”
How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation (2010, background): “[11] compare the annotation quality of sentiment in political blog snippets from a crowdsourcing approach and expert annotators.”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, methodology): “However, most prior research investigates methods for collecting accurate annotations for existing data, for example labeling objects in images or labeling the sentiment of sentences (Hsueh et al., 2009; Liu et al., 2019a; Sun et al., 2020).”
Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data (2016, background): “…empirical studies have found that data created by domain experts can be matched, and sometimes improved at much lower cost, by aggregating judgments of nonexperts (Alonso and Baeza-Yates 2011; Alonso and Mizzaro 2009; Carpenter 2008; Hsueh et al. 2009; Ipeirotis et al. 2013; Snow et al. 2008).”
A Survey of Crowdsourcing Systems (2011, background): “[34] considered the difficult problem of classifying sentiment in political blog snippets.”