Is GPT-3 a Good Data Annotator?

By Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq R. Joty, Boyang Albert Li. Annual Meeting of the Association for Computational Linguistics, 2022

353 citations recorded by the source corpus.

Data annotation is the process of labeling data that could be used to train machine learning models. Having high quality annotation is crucial, as it allows the model to learn the relationship between the input data and the desired output. GPT-3, a large-scale language model developed by OpenAI, has demonstrated im- impressive zero- and few-shot performance on a wide range of NLP tasks. It is therefore natural to wonder whether it can be used to effectively annotate data for NLP tasks. In this paper, we evaluate the performance of GPT-3 as a data annotator by comparing it with traditional data annotation methods and analyzing its output on a range of tasks. Through this analysis, we aim to provide insight into the potential of GPT-3 as a general-purpose data annotator in NLP.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/70b98d90767345b15e0569082c0e4ac661279b5d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, neutral): “Generating labels with large language models (LLMs) is one promising approach, as LLMs have shown a high degree of alignment with human judgment (Gilardi et al., 2023; Ding et al., 2023).”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “On the one hand, LLMs, through extensive pretraining, have acquired a vast repository of knowledge and demonstrate exceptional linguistic comprehension (Kim et al., 2022; Ding et al., 2023a), which forms a foundation for generating faithful data.”
A Survey on In-context Learning (2022, methodology): “For example, Ding et al. (2022) performed a comprehensive analysis and found that generation-based methods are more cost-effective in using GPT-3 than annotating unlabeled data via ICL.”
A Survey for In-context Learning (2023, methodology): “For example, Ding et al. (2022) performed a comprehensive analysis and found that generation-based methods are more cost-effective in using GPT-3 than annotating unlabeled data via ICL.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, background): “It is cited as one of the key drivers of success in modern conversational language models, such as ChatGPT (Liu et al., 2023) and Bard (Manyika, 2023 training language models with reinforcement learning (RL) is that it enables optimization on complex, sequence-level objectives that are not easily…”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “for complex tasks (Ding et al., 2022; Nori et al., 2023; Lee et al., 2023), even though the task of writing biographies is fairly easy.”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “[129] had put forth three distinct prompting approaches for training data generation using GPT-3: unlabeled data annotation, training data generation, and assisted training data generation.”