The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation

By Marzena Karpinska, Nader Akoury, Mohit Iyyer. 2021

136 citations recorded by the source corpus.

Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.

Subfields: Artificial Intelligence, Communication, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “For example, expert evaluators like English teachers (Karpinska et al., 2021) or trained, in-person evaluators (Ippolito et al.”
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “As LLMs become widely adopted in open-ended scenarios involving interaction with humans (e.g., chatbot), many have considered human evaluation using domain experts or crowd raters such as Amazon Mechanical Turk [23, 46] to examine models’ response quality.”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “All the methods are run in parallel, and we run the data collection at the same time to control for the effects of time on data quality (Karpinska et al., 2021).”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “Concurrent work has shown that human evaluation might not always be consistent [10, 29]; however human judgments continue to be the gold standard for evaluating open-ended text generation.”
Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, background): “[56] highlight the perils of using crowdsourced workers for evaluating open-ended generated text, because crowdworkers do not read text as carefully as expert teachers.”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “Note that we used the MTurk Sandbox interface only — no MTurk workers are recruited in our human study due to poor annotation quality for open-ended text generation (Karpinska et al., 2021; Clark et al., 2021).”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “This presents difficulties in conducting reliable and reproducible comparisons between systems (Karpinska et al., 2021), another factor contributing to the popularity of automatic metrics.”
Can Large Language Models Be an Alternative to Human Evaluations? (2023, methodology): “We select open-ended story generation as an example because Karpinska et al. (2021) show that workers from Amazon Mechanical Turk (AMT) cannot distinguish GPT-2 (Radford et al., 2019) generated and human-written stories, 1The WritingPrompts subreddit explicitly forbids the users to use AI for…”
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024, background): “In addition to closed-ended questions, benchmarks can include open-ended questions that are evaluated by human judgment, which can be rated by experts or crowd workers such as Amazon Mechanical Turk (Karpinska et al., 2021; Geng et al., 2023; Wang et al., 2023).”
Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously (2023, background): “…two sets of MOS evaluation: the first one allows all the workers on Mturk to participate in the task and the second one only recruits workers that have HIT Approval Rate ≥ 95% and Number of HITs Approved ≥ 1000 ; these numbers are set based on prior works that conduct human evaluations [15].”
Can Large Language Models Transform Computational Social Science? (2023, background): “In CSS, human evaluations can be unreliable (Karpinska et al., 2021).”
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022, background): “Without proper training, a layperson might struggle to adequately assess generated text (Clark et al., 2021; Karpinska et al., 2021).”
Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “However, this is neither scalable nor reproducible (Karpinska et al., 2021).”
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (2023, background): “In the human evaluation processes, relying on a single perspective can introduce bias and instability in the results (Karpinska et al., 2021).”
Human heuristics for AI-generated language are flawed (2022, methodology): “To explore the effect of increased effort (23), we offered half of the participants in the dating context a bonus payment if they rated at least 75% of the self-presentations correctly.”