Improving language models by retrieving from trillions of tokens

By Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark and 18 more. 2021

1,785 citations recorded by the source corpus.

We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.

Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Quoc V. Le
  6. Graham Neubig
  7. Yejin Choi
  8. Yoshua Bengio

In Retrieval-Augmented Generation

  1. Hannaneh Hajishirzi
  2. Sebastian Riedel
  3. Akari Asai
  4. Danqi Chen
  5. Luyu Gao
  6. Jason Aaron Edward Weston
  7. Jamie Callan
  8. Matei A. Zaharia

In Information Retrieval & Search

  1. Sebastian Riedel
  2. Hannaneh Hajishirzi
  3. Danqi Chen
  4. Gautier Izacard
  5. Wen-tau Yih
  6. Jimmy J. Lin
  7. Edouard Grave
  8. Jamie Callan

Citations in the corpus