Few-shot Learning with Retrieval Augmented Language Models

By Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, Edouard Grave. Journal of machine learning research, 2022

1,295 citations recorded by the source corpus.

Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Prompting & Reasoning

View Paper: https://arxiv.org/abs/2208.03299

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Graham Neubig
  6. Hannaneh Hajishirzi
  7. Yejin Choi
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Mohit Bansal
  5. Xiaogang Wang
  6. Andrew Zisserman
  7. Ming-Hsuan Yang
  8. Jitendra Malik

In Retrieval-Augmented Generation

  1. Luyu Gao
  2. Hannaneh Hajishirzi
  3. Jamie Callan
  4. Danqi Chen
  5. Akari Asai
  6. Wen-tau Yih
  7. Matei A. Zaharia
  8. Edouard Grave

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Tushar Khot
  3. Ashish Sabharwal
  4. Hannaneh Hajishirzi
  5. Timo Schick
  6. Denny Zhou
  7. Zhiyuan Liu
  8. Minlie Huang

Citations in the corpus