How Many Data Samples is an Additional Instruction Worth?

By Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar, Chitta Baral. 2022

21 citations recorded by the source corpus.

Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state-of-the-art task-specific models. Conventional approaches to improve model performance via creating datasets with large number of task instances or architectural changes in the model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augmentation helpful? We augment a subset of tasks in the expanded version of NATURAL INSTRUCTIONS with additional instructions and find that it significantly improves model performance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to ~200 data samples on average across tasks.

Subfields: Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

Citations in the corpus

Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A series of works have found evidence that vanilla language models can be effective at following general language instructions if tuned with annotated “instructional” data – datasets containing language instructional commands and their desired outcome based on human judgement (Weller et al., 2020; M”
Scaling Instruction-Finetuned Language Models (2022, methodology): “Prior work has either scaled the number of templates (Puri et al., 2022), the number of tasks (to 1.”
LILA: A Unified Benchmark for Mathematical Reasoning (2022, result): “This is consistent with the findings of Puri et al. (2022) on instruction-example equivalence.”
BioTABQA: Instruction Learning for Biomedical Table Question Answering (2022, methodology): “[29] introduced instruction augmentation and Prasad et al.”
Is a Question Decomposition Unit All We Need? (2022, background): “…instructional prompts have been pivotal in improving the performance of LMs and achieving zero-shot generalization (Mishra et al., 2021b; Wei et al., 2021; Sanh et al., 2021b; Wei et al., 2022; Ouyang et al., 2022; Parmar et al., 2022; Puri et al., 2022; Kuznia et al., 2022; Luo et al., 2022).”
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models (2022, background): “Prompts and Instructions are shown to be helpful in low-resource settings (Le Scao and Rush, 2021; Puri et al., 2022).”