Instruction Induction: From Few Examples to Natural Language Task Descriptions

By Or Honovich, Uri Shaham, Samuel R. Bowman, Omer Levy. 2022

211 citations recorded by the source corpus.

Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning. We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples. To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction. We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance. This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “First, we use an LLM as an inference model (Ellis et al., 2021; Honovich et al., 2022) to generate instruction candidates based on a small set of demonstrations in the form of input-output pairs.”
A Survey on In-context Learning (2022, background): “1) Instruction Induction (Honovich et al., 2022), APE (Zhou et al.”
Automatic Prompt Optimization with "Gradient Descent" and Beam Search (2023, background): “Furthermore, such works often focus on generating prompts from scratch (Honovich et al., 2022) while it is trivial for humans to write a quick first draft (with e.”
Large language models are few-shot clinical information extractors (2022, methodology): “Across all datasets, similar to Honovich et al. (2022), we assume the latest generation of models on the OpenAI API is the set of InstructGPT models from Ouyang et al. (2022).”
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (2023, methodology): “Instruction Induction tasks also show a wide range of difficulty, resulting in varied settings to be Table 4: Examples of atomic changes’ impact on accuracy using prefix matching (probability ranking shown in Table 2).”
A Survey for In-context Learning (2023, unknown): “…Q-Learning (Zhang et al., 2022a) Ordering(§5.1.2) GlobalE&LocalE (Lu et al., 2022) Formatting (§5.2) Instruction(§5.2.1) Instruction Induction (Honovich et al., 2022), APE (Zhou et al., 2022b), Boostrapping (Wang et al., 2022b) (§5.2.2) CoT (Wang et al., 2022b), Complexity (Fu et al., 2022),…”
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026, unknown): “We instantiate c k with retrieved / randomly sampled few-shot examples or induced high-level instructions [14].”