A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality

By Kenji Hata, Ranjay Krishna, Fei-Fei Li, Michael S. Bernstein. 2016

45 citations recorded by the source corpus.

Microtask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks, making it necessary to understand their behavior over these long periods of time. We utilize three large, longitudinal datasets of nine million annotations collected from Amazon Mechanical Turk to examine claims that workers fatigue or satisfice over these long periods, producing lower quality work. We find that, contrary to these claims, workers are extremely stable in their quality over the entire period. To understand whether workers set their quality based on the task's requirements for acceptance, we then perform an experiment where we vary the required quality for a large crowdsourcing task. Workers did not adjust their quality based on the acceptance threshold: workers who were above the threshold continued working at their usual quality level, and workers below the threshold self-selected themselves out of the task. Capitalizing on this consistency, we demonstrate that it is possible to predict workers' long-term quality using just a glimpse of their quality on the first five tasks.

Subfields: Computer Science Applications, Ocean Engineering, Transportation, AI Ethics, Fairness & Society, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

1. Vittorio Ferrari
2. Ranjay Krishna
3. Li Fei-Fei
4. Sharon Zhou
5. Mitchell L. Gordon
6. Austin Narcomey
7. Michael S. Bernstein

In Ocean Engineering

1. Li Fei-Fei

In Transportation

1. Michael S. Bernstein
2. Li Fei-Fei

In AI Ethics, Fairness & Society

1. Michael S. Bernstein

In Human-Computer Interaction

1. Vittorio Ferrari
2. Jasper R. R. Uijlings
3. Michael S. Bernstein
4. Dim P. Papadopoulos
5. Li Fei-Fei
6. Michael Gygli

Citations in the corpus

• Extreme Clicking for Efficient Object Annotation (2017, background): “Annotators that have reached the desired level of quality can be expected to keep it throughout the annotation [30].”
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, methodology): “We employ three blocks, to balance quality estimates against evaluators’ fatigue [32, 50, 22].”