How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation

By Stefanie Nowak, Stefan M. Rüger. Multimedia Information Retrieval, 2010

414 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4e227d4919caa1780ec2cd67e834430037e294bb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Computer Vision and Pattern Recognition

Citations in the corpus

Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data (2016, methodology): “Given the open nature of the platform, it is vital to prevent them from participating in a job, using careful screening and quality control (e.g., Berinsky et al. 2014; Eickhoff and de Vries 2012; Kapelner and Chandler 2010; Nowak and Rger 2010;).”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, background): “In the other NLP domains, crowdsourcing has been proposed as an alternative to overcome these challenges, showing that crowd workers’ aggregated responses could produce quality approaching those produced by experts (Snow et al., 2008; Callison-Burch, 2009; Nowak and R¨uger, 2010).”
CDAS: A Crowdsourcing Data Analytics System (2012, background): “Typical tasks include image annotation [21][18], information retrieval [1][8] and natural language processing [3][12][17].”
How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy (2011, background): “On interpretative tasks, different backgrounds, interests, and theoretical perspectives can lead to different interpretation of the data and to a substantial variation between annotators [13,14].”
A Survey of Crowdsourcing Systems (2011, background): “In 2010, Nowak and Rüger [63] conducted a study about inter-annotator agreement for multi-label image annotation.”
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2021, methodology): “Obtaining multiple annotator judgements on the same data instances is a common practice in NLP in order to improve the quality of ﬁnal labels (Snow et al., 2008; Nowak and Rüger, 2010).”