REPLUG: Retrieval-Augmented Black-Box Language Models

By Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Michael Lewis, Luke S. Zettlemoyer, Wen-tau Yih. 2023

1,051 citations recorded by the source corpus.

We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%. Code is publicly released at github.com/swj0419/REPLUG.

Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Graham Neubig
  6. Yejin Choi
  7. Hannaneh Hajishirzi
  8. Noah A. Smith

In Retrieval-Augmented Generation

  1. Luyu Gao
  2. Jamie Callan
  3. Danqi Chen
  4. Hannaneh Hajishirzi
  5. Wen-tau Yih
  6. Luke S. Zettlemoyer
  7. Edouard Grave
  8. Jimmy J. Lin

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Timo Schick
  3. Mohit Bansal
  4. Jianfeng Gao
  5. Furu Wei
  6. Hinrich Schütze
  7. Graham Neubig
  8. Weizhu Chen

Citations in the corpus