REPLUG: Retrieval-Augmented Black-Box Language Models
By Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Michael Lewis, Luke S. Zettlemoyer, Wen-tau Yih. 2023
1,051 citations recorded by the source corpus.
We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%. Code is publicly released at github.com/swj0419/REPLUG.
Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
In Retrieval-Augmented Generation
- Luyu Gao
- Jamie Callan
- Danqi Chen
- Hannaneh Hajishirzi
- Wen-tau Yih
- Luke S. Zettlemoyer
- Edouard Grave
- Jimmy J. Lin
In Large Language Models
- Luke S. Zettlemoyer
- Timo Schick
- Mohit Bansal
- Jianfeng Gao
- Furu Wei
- Hinrich Schütze
- Graham Neubig
- Weizhu Chen
Citations in the corpus
- Augmented Language Models: a Survey (2023, background): “…by adding the retrieved documents to the current context (Chen et al., 2017; Clark and Gardner, 2017; Lee et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020; Zhong et al., 2022; Borgeaud et al., 2022; Izacard et al., 2022; Shi et al., 2023).”
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, positive): “To address this problem, retrieval-augmented LLMs (Borgeaud et al., 2022; Izacard et al., 2023; Shi et al., 2023), which incorporate non-parametric knowledge into LLMs with additional retrieval modules, have gained increasing attention.”
- Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, background): “…considerable interest in retrieval-1 The GTE model is publicly available at https:// huggingface.co/thenlper/gte-large augmented systems based on text embedding models that integrate the reasoning and comprehension capabilities of LLMs (Izacard et al., 2022b; Ram et al., 2023; Shi et al., 2023).”
- Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “Multimodal dialog is enabled with image-sharing and image-response behaviors. techniques (Shuster et al., 2021; Ram et al., 2023; Shi et al., 2023), there is still a need for thorough evaluation of their efficacy in handling very long conversations.”
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “…retriever is also a critical component for retrieval-augmented generation (RAG) (Lewis et al., 2020), which allows LLMs to access the most up-to-date external or proprietary knowledge without modifying the model parameters (Liu et al., 2024; Guu et al., 2020; Shi et al., 2023; Wang et al., 2023a).”
- C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, background): “…not only for its wide usage in typical applications, like web search and question answering (Karpukhin et al., 2020) but also due to its fundamental role in augmenting large language models (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022; Shi et al., 2023).”
- The Faiss Library (2024, positive): “Finally, [81], [82], [83] and [84] consist of language models that have been trained to integrate textual retrieval in order to improve their accuracy, factuality or compute efficiency.”
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, positive): “LD-Agent and HippoRAG extract additional information and leverage them to better pinpoint entity-centric knowledge within the index.”
- Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, methodology): “By using external knowledge as guidance, retrieval-augmented models can generate more accurate and reliable responses (Guu et al. 2020; Lewis et al. 2020; Borgeaud et al. 2022; Izacard et al. 2022; Shi et al. 2023; Ren et al. 2023 The process of data generation.”
- Adapting Language Models to Compress Contexts (2023, methodology): “However, REPLUG top-10 outperforms Fused Summaries.”
- Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “By aligning the retriever’s capabilities with the preferences of the LLMs through feedback signals, both can be better coordinated [ Yu et al. , 2023b, Izacard et al. , 2022, Yang et al. , 2023b, Shi et al. , 2023 ] .”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “REPLUG [127] illustrates this methodology by treating the language model as a “black box”, utilizing Contriever to seamlessly incorporate relevant external documents into the query.”
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023, background): “As language models have become increasingly sophisticated, there is a growing trend towards in-context joint retrieval and generation, treating the LLM as a fixed black box (Levine et al., 2022; Ram et al., 2023; Shi et al., 2023).”
- A survey on large language model based autonomous agents (2023, neutral): “Model Profile Memory Planning Action LS Time Operation Structure WebGPT[105] - 2 2 12/2021 SayCan[2] 1 2 3 04/2022 MRKL[71] 1 2 - 05/2022 Inner Monologue[64] 2 2 3 07/2022 Social Simulacra[110] 2 - 1 - 08/2022 ReAct[151] 2 2 3 10/2022 REPLUG[124] 2 1 - 1 - 01/2023 MALLM[121] 2 2 - 1 - 01/2023 DEPS[1”
- Large Language Models as Analogical Reasoners (2023, background): “LLMs with billions of parameters demonstrate in-context learning and few-shot learning abilities (Brown et al., 2020; Liu et al., 2022; Su et al., 2022; Mishra et al., 2022; Wei et al., 2022a; Yasunaga et al., 2023; Shi et al., 2023).”
- In-Context Retrieval-Augmented Language Models (2023, background): “In a concurrent work, Shi et al. (2023) also suggest to augment off-the-shelf LMs with retrieved texts by prepending them to the input.”