Improving language models by retrieving from trillions of tokens
By Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark and 18 more. 2021
1,785 citations recorded by the source corpus.
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.
Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Information Retrieval & Search
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yejin Choi
- Yoshua Bengio
In Retrieval-Augmented Generation
- Hannaneh Hajishirzi
- Sebastian Riedel
- Akari Asai
- Danqi Chen
- Luyu Gao
- Jason Aaron Edward Weston
- Jamie Callan
- Matei A. Zaharia
In Information Retrieval & Search
- Sebastian Riedel
- Hannaneh Hajishirzi
- Danqi Chen
- Gautier Izacard
- Wen-tau Yih
- Jimmy J. Lin
- Edouard Grave
- Jamie Callan
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “This approach effectively increases the number of data tokens seen during training (by a factor of ∼ 10 in Borgeaud et al. (2021)).”
- PaLM: Scaling Language Modeling with Pathways (2022, background): “One area is retrieval models that aim to drastically reduce model sizes by embedding large amounts of text the model can have access to later (Guu et al., 2020; Borgeaud et al., 2021).”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Lazaridou et al. (2022) explore NaturalQuestions in a 15-shot setup using Gopher, augmenting questions with 50 passages retrieved using Google Search.”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…information during pretraining, including various forms of metadata (Keskar et al., 2019), HTML tags (Aghajanyan et al., 2021), Wikipedia markup (Schick et al., 2022), or related texts obtained from an information retrieval system (Guu et al., 2020; Borgeaud et al., 2021; Izacard et al., 2022).”
- LaMDA: Language Models for Dialog Applications (2022, background): “The RETRO architecture demonstrates that language models can be primed with results retrieved from a database as large as two trillion tokens [41].”
- Competition-level code generation with AlphaCode (2022, background): “While such copying can be beneficial (Borgeaud et al., 2021), referring to information that was not available at the time of the competition would constitute cheating.”
- Emergent Abilities of Large Language Models (2022, background): “4Though note that encoding world knowledge in parameters is just one approach; there are others (e.g., Guu et al., 2020; Borgeaud et al., 2021).”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “We aim to be fully accountable for the development lifecycle of OPT-175B, and only through increasing transparency around LLM development can we start understanding the limitations and risks of LLMs before broader deployment occurs.”
- Galactica: A Large Language Model for Science (2022, methodology): “From Table 10, Galactica can compose its knowledge into the question-answering task, and performance is strong; significantly outperforming the other open language models, and outperforming a larger model (Gopher 280B) in the majority of tasks.”
- A Review of Sparse Expert Models in Deep Learning (2022, methodology): “But in light of the recent scaling results from Hoff-mann et al. (2022) which recommends more tokens to train compute-optimal models (Chinchilla was a 70B parameter model trained on 1.4T tokens), future work might revisit this analysis.”
- Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Recently, RETRO (Borgeaud et al., 2021) used retrieval over a database of tril-lions of tokens.”
- Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “language models have been recently gaining momentum (Khandelwal et al., 2019; Guu et al., 2020; Yogatama et al., 2021; Borgeaud et al., 2021), extending monolithic parametric models with information from a knowledge source.”
- Improving alignment of dialogue agents via targeted human judgements (2022, background): “This mechanism also allows for temporal generalisation beyond a static parametric model (Borgeaud et al., 2022; Lewis et al., 2020; Liška et al., 2022; Shuster et al., 2022a).”
- PEER: A Collaborative Language Model (2022, background): “It would thus be interesting to investigate how incorporating a retrieval engine that does not have access to xt+1 or even jointly training it along with the model, as is done by Guu et al. (2020) and Borgeaud et al. (2021), would affect results.”
- TALM: Tool Augmented Language Models (2022, background): “RETRO [Borgeaud et al., 2021] adds an ”internet scale” retrieval mechanism.”
- Predictability and Surprise in Large Generative Models (2022, background): “g, How do you ensure the system does not generate inappropriate and harmful outputs, such as making overtly sexist or racist comments [65]? How do you identify bias issues in the system prior to deployment [8, 53]? How do you ensure that when the model outputs a claim, it isn’t making up facts [10]?”
- Teaching language models to support answers with verified quotes (2022, background): “Many Question-Answering papers have developed “deep learning”-based retrieval systems with KNN lookups (Borgeaud et al., 2021; Guu et al., 2020; Lewis et al., 2021a).”
- Learning To Retrieve Prompts for In-Context Learning (2021, methodology): “Retrieval methods have also been successfully used in language modeling (Khandelwal et al., 2020; Borgeaud et al., 2021; Alon et al., 2022) and machine translation (Khandelwal et al., 2021).”