Selective Annotation Makes Language Models Better Few-Shot Learners

By Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke S. Zettlemoyer, Noah A. Smith and 1 more. International Conference on Learning Representations, 2022

329 citations recorded by the source corpus.

Many recent approaches to natural language tasks are built on the remarkable abilities of large language models. Large language models can perform in-context learning, where they learn a new task from a few task demonstrations, without any parameter updates. This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time. Based on this framework, we propose an unsupervised, graph-based selective annotation method, voke-k, to select diverse, representative examples to annotate. Extensive experiments on 10 datasets (covering classification, commonsense reasoning, dialogue, and text/code generation) demonstrate that our selective annotation method improves the task performance by a large margin. On average, vote-k achieves a 12.9%/11.4% relative gain under an annotation budget of 18/100, as compared to randomly selecting examples to annotate. Compared to state-of-the-art supervised finetuning approaches, it yields similar performance with 10-100x less annotation cost across 10 tasks. We further analyze the effectiveness of our framework in various scenarios: language models with varying sizes, alternative selective annotation methods, and cases where there is a test data domain shift. We hope that our studies will serve as a basis for data annotations as large language models are increasingly applied to new tasks. Our code is available at https://github.com/HKUNLP/icl-selective-annotation.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/86d0d3855f94105e25d81cab9f3d269c6062a9c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (2023, methodology): “To ensure the diversity and representativeness of topic distribution in selected prompts, we adopt the similar method introduced in [56] for selection.”
Large Language Models as Analogical Reasoners (2023, background): “LLMs with billions of parameters demonstrate in-context learning and few-shot learning abilities (Brown et al., 2020; Liu et al., 2022; Su et al., 2022; Mishra et al., 2022; Wei et al., 2022a; Yasunaga et al., 2023; Shi et al., 2023).”
Automatic Chain of Thought Prompting in Large Language Models (2022, methodology): “…to the test instance where the popular practice is dynamically retrieving related training examples for a given test input [Rubin et al., 2022, Su et al., 2022]; (ii) augmenting with fine-grained information, such as incorporating task instruction [Mishra et al., 2022, Wei et al., 2022b, Sanh…”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “Retrieval methods can be generally categorized into two types: sparse and dense, based on the information encoding methods.”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, methodology): “Text Embeddings Text embeddings are useful in many applications such as information retrieval (Thakur et al., 2021), text similarity (Gao et al., 2021), prompt retrieval for in-context learning (Su et al., 2022), classification (Reimers and Gurevych, 2019), and beyond.”
Complexity-Based Prompting for Multi-Step Reasoning (2022, background): “Example Selection for Prompting Designing prompts can be challenging due to the instability, as multiple works have shown the performance is sensitive to prompt, task, dataset, and model changes (Zhao et al., 2021; Lu et al., 2022; Su et al., 2022).”
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, methodology): “0 is a deterministic process, using TIFA to create a new benchmark might not be deterministic as the underlying question generator (GPT-3 in our case) might change privately.”
A Survey of Large Language Models (2023, unknown): “To resolve this issue, diversitybased selection strategies are proposed to choose the most representative set of examples for specific tasks [222, 223].”
A Survey on In-context Learning (2022, background): “Lu et al. (2022) defined the global and local entropy metrics.”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, unknown): “To address this issue, Sudalairaj et al. (2024) opt for selecting examples that concentrate on specific aspects to better stimulate the long tail of knowledge inherent in LLMs. Liu et al. (2022b) and Su et al. (2023) prioritize consistent samples as demonstra-tive examples based on their cosine simi”
Reasoning with Language Model Prompting: A Survey (2022, unknown): “Calibrator (Ye and Durrett, 2022), Human-AI (Wiegreffe et al., 2022) Ensemble-Optimization Self-C (Wang et al., 2022e), DIVERSE (Li et al., 2022d), Complexity (Fu et al., 2023b), Self-V (Weng et al., 2022), MCR (Yoran et al., 2023) Iterative-Optimization STaR (Zelikman et al., 2022), LMSI (Huang et ”
Binding Language Models in Symbolic Languages (2022, methodology): “On TABFACT, we further annotate a pool of 200 examples with BINDER programs from the training set using vote-k selective annotation (Su et al., 2022), and use them by retrieving relevant few-shot examples for each inference example via maximum inner-product similarity of the SentenceBert (Clark et a”