Want To Reduce Labeling Cost? GPT-3 Can Help

By Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, Michael Zeng. Conference on Empirical Methods in Natural Language Processing, 2021

338 citations recorded by the source corpus.

Data annotation is a time-consuming and labor-intensive process for many NLP tasks. Although there exist various methods to produce pseudo data labels, they are often task-specific and require a decent amount of labeled data to start with. Recently, the immense language model GPT-3 with 175 billion parameters has achieved tremendous improvement across many few-shot learning tasks. In this paper, we explore ways to leverage GPT-3 as a low-cost data labeler to train other models. We find that, to make the downstream model achieve the same performance on a variety of NLU and NLG tasks, it costs 50% to 96% less to use labels from GPT-3 than using labels from humans. Furthermore, we propose a novel framework of combining pseudo labels from GPT-3 with human labels, which leads to even better performance with limited labeling budget. These results present a cost-effective data labeling methodology that is generalizable to many practical applications.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4e263b4cd6998bff2501dd143e685f413179b12d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “It supports learning from limited labeled data, since the larger teacher model is often used to generate a training dataset with noisy pseudo labels (Chen et al., 2020; Il-iopoulos et al., 2022; Wang et al., 2021; Smith et al., 2022a; Arora et al., 2022; Agrawal et al., 2022).”
Is GPT-3 a Good Data Annotator? (2022, methodology): “Wang et al. (2021) investigate methods to utilize GPT-3 to annotate unlabeled data.”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, background): “However, these works typically impose a strong assumption on the teacher LM (Wang et al., 2021), and require manually constructed set of prompts (Bhagavatula et al., 2022).”
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2023, methodology): “LLMs for Data Generation Many research studies have employed LLMs for data generation, focusing on various tasks such as question answering (Wang et al. 2021; Agrawal et al. 2022; Chen, Chen, and tau Yih 2023), semantic similarity predictions (Schick and Sch¨utze 2021), and instruction tuning…”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, neutral): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
A Survey for In-context Learning (2023, background): “Combining pseudo labels from GPT-3 with human labels leads to even better performance at a small cost (Wang et al., 2021b).”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “Selective annotation represents an optimal tradeoff between expensive and precise human annotation and economic but relatively rough LLMs-based annotation(Wang et al., 2021; Kocon et al., 2023).”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, neutral): “Prior work has explored the use of language models to generate training data in the few-shot setting (Wang et al., 2021).”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “Collecting high-quality labeled data for new tasks and languages is expensive and resource-consuming (Wang et al., 2021).”
A Survey on In-context Learning (2022, background): “Combining pseudo labels from GPT-3 with human labels leads to even better performance at a small cost (Wang et al., 2021).”
Co-training Improves Prompt-based Learning for Large Language Models (2022, background): “Wang et al. (2021) propose to train smaller models on labels from GPT-3 to reduce annotation cost, but they train from individual, uncalibrated prompts and do not attempt to reﬁne the prompt model alongside the smaller model.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
Large language models are few-shot clinical information extractors (2022, methodology): “To remedy these issues, we follow several recent works (Lang et al., 2022a; Smith et al., 2022; Wang et al., 2021) and show that we can instead view the LLM + resolver system as a labeler rather than as a classifier , and that this can even boost performance.”
Uncertainty Estimation for Language Reward Models (2022, background): “While prior work has attempted to leverage language models to automate part of the labeling process [48, 25], we are not aware of any work ﬁne-tuning language models that has tried actively selecting points for human feedback.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “In the context of NLU tasks, certain studies (Ye et al., 2022; Gao et al., 2023a; Wang et al., 2021a) explore the use of labels as meta-information to synthesize corresponding samples for data augmentation.”