In-Context Retrieval-Augmented Language Models

By Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham. Transactions of the Association for Computational Linguistics, 2023

1,053 citations recorded by the source corpus.

Abstract Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.1

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/465471bb5bf1a945549d6291c2d23367966b4957

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Comprehensive Overview of Large Language Models (2023, positive): “This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “The most straightforward approach to utilize retrieved knowledge or tool feedback is to directly concatenate them with user queries before prompting LLMs (Shi et al., 2023c; Mallen et al., 2023; Ram et al., 2023).”
MemGPT: Towards LLMs as Operating Systems (2023, methodology): “…generated response against the ‘gold response’ using ROUGE-L scores (Lin, 2004) and an ‘LLM judge’, which is instructed to evaluate whether or not the generated response is consistent with the gold response (GPT-4 has been shown to have high agreement with human evaluators (Zheng et al., 2023)).”
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, background): “Along with the emergence of LLMs with superior reasoning capabilities thanks to their billion-sized parameters (Wei et al., 2022a), a synergy between LLMs and retrievers has led to significant advancements (Lazaridou et al., 2022; Ram et al., 2023).”
RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, methodology): “…al., 2022), recent work has suggested that simply adding retrieved documents to the input of a standard LM can also work well (Khattab et al., 2022; Ram et al., 2023; Shi et al., 2023), thus making it possible to use retrieval-augmented strategies in combination with LLMs that are only available…”
ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, background): “In addition, it has been shown that off-the-shelf general purpose retrievers can improve a baseline language model significantly without further fine-tuning [48].”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, methodology): “For RALMs, we follow the definition of in-context RALMs from Ram et al. (2023), where context sentences are retrieved from a corpus C , and generation is conditioned on the retrieved context.”
LightRAG: Simple and Fast Retrieval-Augmented Generation (2024, neutral): “As large language models (LLMs) evolve, researchers aim to enhance their ability to interpret graph-structured data.”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “Multimodal dialog is enabled with image-sharing and image-response behaviors. techniques (Shuster et al., 2021; Ram et al., 2023; Shi et al., 2023), there is still a need for thorough evaluation of their efficacy in handling very long conversations.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, background): “…considerable interest in retrieval-1 The GTE model is publicly available at https:// huggingface.co/thenlper/gte-large augmented systems based on text embedding models that integrate the reasoning and comprehension capabilities of LLMs (Izacard et al., 2022b; Ram et al., 2023; Shi et al., 2023).”
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “Retrieval-Augmented Generation (RAG) methods (Figure 1 left; Lewis et al. 2020; Guu et al. 2020) augment the input of LLMs with relevant retrieved passages, reducing factual errors in knowledge-intensive tasks (Ram et al., 2023; Asai et al., 2023a).”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “REALM and RAG do not use LLMs, but their frameworks have been widely applied in many following approaches.”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, methodology): “3) Not Grounding on External Knowledge: Our method relies solely on the model’s internal knowledge and does not use external retrieval modules like some retrieval augmented LMs do (Izacard et al., 2022; Borgeaud et al., 2022; Ram et al., 2023).”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, methodology): “As an alternative to kNN-LM, RIC-LM (Shi et al., 2023; Ram et al., 2023) retrieves text blocks from a datastore and feeds them to the parametric LM in context.”
RAFT: Adapting Language Model to Domain Specific RAG (2024, background): “(Shi et al., 2023d; Ram et al., 2023) assume black-box access to an LM and combine it with either off-the-shelf or fine-tuned retriever.”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “…progressed from simple token [ Khandelwal et al. , 2019 ] and entity retrieval [ Nishikawa et al. , 2022 ] to more complex structures like chunks [ Ram et al. , 2023 ] and knowledge graph [ Kang et al. , 2023 ] , with studies focusing on the granularity of retrieval and the level of data…”