A Survey of NLP-Related Crowdsourcing HITs: what works and what does not

By Jessica Huynh, Jeffrey P. Bigham, Maxine Eskénazi. 2021

21 citations recorded by the source corpus.

Crowdsourcing requesters on Amazon Mechanical Turk (AMT) have raised questions about the reliability of the workers. The AMT workforce is very diverse and it is not possible to make blanket assumptions about them as a group. Some requesters now reject work en mass when they do not get the results they expect. This has the effect of giving each worker (good or bad) a lower Human Intelligence Task (HIT) approval score, which is unfair to the good workers. It also has the effect of giving the requester a bad reputation on the workers' forums. Some of the issues causing the mass rejections stem from the requesters not taking the time to create a well-formed task with complete instructions and/or not paying a fair wage. To explore this assumption, this paper describes a study that looks at the crowdsourcing HITs on AMT that were available over a given span of time and records information about those HITs. This study also records information from a crowdsourcing forum on the worker perspective on both those HITs and on their corresponding requesters. Results reveal issues in worker payment and presentation issues such as missing instructions or HITs that are not doable.

Subfields: Computer Science Applications, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Here we describe the annotation instructions for our analysis of 66 ACL, EMNLP, and INLG papers from 2021.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “Careful instructions (Huynh et al., 2021) can result in good quality feedback or labels to learn from; however, the distribution of data, which is typically decided by those instructions, is unlikely to match the changing desires of organic users, and takes significant resources to collect.”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “However, the use of crowdworkers itself has a number of pitfalls to avoid as well (Huynh et al., 2021).”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “While careful instructions (Huynh et al., 2021) can result in good quality feedback or labels to learn from, collection both involves signiﬁcant monetary costs – where anno-tators should be paid well above minimum wage – and the pool of workers may be limited (Moss et al., 2020).”