Few-shot Learning with Retrieval Augmented Language Models
By Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, Edouard Grave. Journal of machine learning research, 2022
1,295 citations recorded by the source corpus.
Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Prompting & Reasoning
View Paper: https://arxiv.org/abs/2208.03299
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Hannaneh Hajishirzi
- Yejin Choi
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Mohit Bansal
- Xiaogang Wang
- Andrew Zisserman
- Ming-Hsuan Yang
- Jitendra Malik
In Retrieval-Augmented Generation
- Luyu Gao
- Hannaneh Hajishirzi
- Jamie Callan
- Danqi Chen
- Akari Asai
- Wen-tau Yih
- Matei A. Zaharia
- Edouard Grave
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Tushar Khot
- Ashish Sabharwal
- Hannaneh Hajishirzi
- Timo Schick
- Denny Zhou
- Zhiyuan Liu
- Minlie Huang
Citations in the corpus
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Specifically, we use Atlas (Izacard et al., 2022), a retrieval-augmented LM finetuned on Natural Questions (Kwiatkowski et al.”
- Augmented Language Models: a Survey (2023, methodology): “Various works augment LMs with a dense retriever by adding the retrieved documents to the current context (Chen et al., 2017; Clark and Gardner, 2017; Lee et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020; Zhong et al., 2022; Borgeaud et al., 2022; ”
- Galactica: A Large Language Model for Science (2022, methodology): “Examples of such models include RAG, RETRO and Atlas (Lewis et al., 2020b; Borgeaud et al., 2021; Izacard et al., 2022).”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “Atlas [13] further investigates the effect of various losses on the performance of the retriever.”
- Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, methodology): “Similarly, Izacard et al. (2022b) observed using the latest version of Wikipedia (12 / 2021) could lead to 4.4 drops of the EM score, compared to the Wikipedia version (12 / 2018) that the NQ questions are created from.”
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, methodology): “3) Not grounding on external knowledge: Our method relies on the model’s internal knowledge without using external retrieval modules (Izacard et al., 2022; Borgeaud et al., 2022; Ram et al., 2023).”
- Nomic Embed: Training a Reproducible Long Context Text Embedder (2024, background): “Text embeddings are an integral component of modern NLP applications powering retrieval-augmented-generation (RAG) for LLMs and semantic search (Lewis et al., 2021a; Izacard et al., 2022b; Ram et al., 2023).”
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, positive): “In-context In-context learning methods now routinely invoke tools, leading to LM pipelines that use retrieval (Chen et al., 2017; Lewis et al., 2020; Guu et al., 2020; Lazaridou et al., 2022; Izacard et al., 2022), and APIs (Nakano et al., 2021).”
- Retrieval-based Language Models and Applications (2023, background): “…there has been growing interest in retrieval-based LMs (Guu et al., 2020; Khan-delwal et al., 2020; Borgeaud et al., 2022; Zhong et al., 2022; Izacard et al., 2022b; Min et al., 2022), which incorporate a non-parametric datastore (e.g., text chunks from an external corpus) with their…”
- Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning (2022, neutral): “We hypothesize that this is due to the ability to incorporate snippets of partially relevant information as explicit contrastive examples, similar to phenomenon observed in few-shot learning and QA in works like Borgeaud et al. (2022); Izacard et al. (2022).”
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023, methodology): “Previous retrieval-augmented LMs Izacard et al. [2022a], Shi et al. [2023] typically adopted one-time retrieval, i.e., to retrieve knowledge using only the task input (e.g., a user question for open-domain question answering). One-time retrieval should suffice to fulfill the information needs if the”
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, background): “…augmenting LMs with non-parametric memories (i.e., retrieved text chunks) enables much smaller models to match the performance of larger models (Izacard et al., 2022b; Khandelwal et al., 2020; Min et al., 2022), although Chen et al. (2022) and Longpre et al. (2021) show that even those models…”
- Enabling Large Language Models to Generate Text with Citations (2023, background): “Retrieval-augmented LMs (Borgeaud et al., 2022; Izacard et al., 2022) incorporate retrieved passages during both training and inference, but do not guarantee faithfulness to retrieved passages or explicitly provide citations.”
- FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2023, positive): “Our F RESH P ROMPT method leverages a text prompt to (1) introduce contextually relevant and up-to-date information (including answers to relevant questions) from a search engine to a pre-trained LLM , and (2) teach the model to reason over retrieved evidences.”
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2023, background): “Retrieval-augmented generation (RAG; Guu et al. 2020; Lewis et al. 2020; Khattab et al. 2021; Izac-ard et al. 2022) is now a common strategy for bol-stering LLMs by combining them with retrieval systems.”
- Tool Learning with Foundation Models (2023, positive): “an end-to-end fashion, achieving superior performance in knowledge-intensive NLP tasks (Guu et al., 2020; Lewis et al., 2020b; Izacard et al., 2022).”