Making Pre-trained Language Models Better Few-shot Learners

By Tianyu Gao, Adam Fisch, Danqi Chen. 2021

2,303 citations recorded by the source corpus.

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF—better few-shot fine-tuning of language models—a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yejin Choi
  6. Graham Neubig
  7. Hannaneh Hajishirzi
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Mohit Bansal
  7. Jitendra Malik
  8. Andrew Zisserman

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Hannaneh Hajishirzi
  5. Timo Schick
  6. Mohit Bansal
  7. Noah A. Smith
  8. Michael Lewis

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Hannaneh Hajishirzi
  3. Timo Schick
  4. Hinrich Schütze
  5. Michael Lewis
  6. Yejin Choi
  7. Denny Zhou
  8. Percy Liang

Citations in the corpus