ChatGPT outperforms crowd workers for text-annotation tasks

By Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli. 2023

1,630 citations recorded by the source corpus.

Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles (n = 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT’s intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003—about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.

Subfields: Artificial Intelligence, Sociology and Political Science, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “We posit that the superior writing abilities of LLMs, as manifested in surpassing human annotators in certain tasks, are fundamentally driven by RLHF, as documented in Gilardi et al. (2023) and Huang et al. (2023).”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “As LLMs continue to improve, they show potential in replacing human annotators in many tasks [14, 17].”
Visual Instruction Tuning (2023, methodology): “Inspired by the success of recent GPT models in text-annotation tasks [17], we propose to leverage ChatGPT/GPT-4 for multimodal instruction-following data collection, based on the widely existing image-pair data.”
A Survey of Large Language Models (2023, neutral): “Since LLMs perform well for text annotation [228], some recent studies employ LLM itself as the demonstration generator without human intervention [229, 230].”
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving (2023, background): “This lanGen enables the transformation of vector representations into human-readable language captions.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Another approach is demonstrated by Gilardi et al. (2023), who employ ChatGPT as an annotator to categorize inputs.”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, neutral): “The development of Vision-Language Models (VLMs) has transitioned from being rooted in BERT-based language decoders [13, 31, 47] for merging visual and textual data [34, 36, 59, 64], to a notable advancement ushered by the integration of Large Language Models (LLMs) [4, 7, 10, 11, 17, 53, 61–63].”
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023, background): “LLMs are believed to be better annotators than human-being in many scenarios (Gilardi et al., 2023).”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, background): “5 [3] and GPT-4 [4] are proven to effectively perform text-annotation tasks [39] or instruct other models to follow instructions of specific domains [40, 35, 12, 37].”
Is GPT-3 a Good Data Annotator? (2022, background): “A study conducted by Gilardi et al. (2023) further illustrates the superior performance of ChatGPT compared to crowd-workers for various annotation tasks.”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, neutral): “Generating labels with large language models (LLMs) is one promising approach, as LLMs have shown a high degree of alignment with human judgment (Gilardi et al., 2023; Ding et al., 2023).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, neutral): “…feedback [Bai et al., 2022b, Saunders et al., 2022, Ye et al., 2023, Kim et al., 2023, Akyürek et al., 2023, Madaan et al., 2023, Chen et al., 2023, Gilardi et al., 2023] with research agendas like Recursive Reward Modeling [Leike et al., 2018] and AI Safety via debate [Irving et al., 2018, Du et…”
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, neutral): “Given the multitude of LLMs trained on various diverse pre-training corpora, creating a test dataset that is genuinely independent and completely untouched by any model is extremely difficult, especially when the pre-trained data of many models remains closed-source.”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “Moreover, several studies (Hosking et al., 2023; Singh et al., 2023; Gilardi et al., 2023) have highlighted that human-generated data, being inherently susceptible to biases and errors, may not even be optimal for model training or evaluation.”
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models (2023, background): “…AlpacaFarm (Dubois et al., 2023) asks the model to select “which response is better based on your judgment and based on your own pref-erence” Another example is recent work that showed ChatGPT can outperform crowd-workers for text-annotation tasks (Gilardi et al., 2023; Chiang & yi Lee, 2023).”
Learning to Compress Prompts with Gist Tokens (2023, methodology): “Recent work has found that ChatGPT can be used for text annotation and evaluation (Gilardi et al., 2023; Huang et al., 2023; Wang et al., 2023) with comparable performance to human annotation, and similar model-based evaluations have been conducted with recent LMs (Taori et al.”