Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

By Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap and 30 more. 2022

1,112 citations recorded by the source corpus.

How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions—training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Jianfeng Gao
  5. Graham Neubig
  6. Yejin Choi
  7. Hannaneh Hajishirzi
  8. Noah A. Smith

In Information Systems

  1. Hannaneh Hajishirzi
  2. Wen-tau Yih
  3. Luke S. Zettlemoyer
  4. Akari Asai
  5. Percy Liang
  6. Wayne Xin Zhao
  7. Ji-rong Wen
  8. Sebastian Riedel

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Yejin Choi
  3. Timo Schick
  4. Hinrich Schütze
  5. Hannaneh Hajishirzi
  6. Noah A. Smith
  7. Michael Lewis
  8. Sebastian Riedel

In Natural Language Processing

  1. Luke S. Zettlemoyer
  2. Mohit Bansal
  3. Graham Neubig
  4. Percy Liang
  5. Yejin Choi
  6. Hannaneh Hajishirzi
  7. Noah A. Smith
  8. Jianfeng Gao

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Hannaneh Hajishirzi
  3. Timo Schick
  4. Michael Lewis
  5. Pengfei Liu
  6. Hongsheng Li
  7. Lei Li
  8. Or Honovich

Citations in the corpus