The Power of Scale for Parameter-Efficient Prompt Tuning

By Brian Lester, Rami Al-Rfou, Noah Constant. 2021

5,785 citations recorded by the source corpus.

In this work, we explore “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3’s few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method “closes the gap” and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant because large models are costly to share and serve and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed “prefix tuning” of Li and Liang (2021) and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “prompt ensembling.” We release code and model checkpoints to reproduce our experiments.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Quoc V. Le
  6. Graham Neubig
  7. Yejin Choi
  8. Yoshua Bengio

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Mohit Bansal
  5. Hannaneh Hajishirzi
  6. Timo Schick
  7. Jianfeng Gao
  8. Michael Lewis

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Hannaneh Hajishirzi
  3. Swaroop Mishra
  4. Denny Zhou
  5. Timo Schick
  6. Yejin Choi
  7. Hinrich Schütze
  8. Mohit Bansal

Citations in the corpus