Retrieval-based Language Models and Applications

By Akari Asai, Sewon Min, Zexuan Zhong, Danqi Chen. 2023

142 citations recorded by the source corpus.

Retrieval-based language models (LMs) have shown impressive performance on diverse NLP tasks. In this tutorial, we will provide a comprehensive and coherent overview of recent advances in retrieval-based LMs. We will start by providing preliminaries covering the foundation of LMs (e.g., masked LMs, autoregressive LMs) and retrieval systems (e.g., nearest-neighbor search). We will then detail recent progress in retrieval-based models, focusing on their model architectures and learning approaches. Finally, we will show how retrieval-based LMs are adapted to downstream applications, and extended to multilingual and multi-modal settings. Finally, we will use an exercise to showcase the effectiveness of retrieval-based LMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Hannaneh Hajishirzi
  2. Akari Asai
  3. Sean Welleck
  4. Yizhong Wang
  5. Luke S. Zettlemoyer
  6. Hailey Schoelkopf
  7. Hong-Liu Cao
  8. Mohit Bansal

In Computer Vision and Pattern Recognition

  1. Hannaneh Hajishirzi
  2. Akari Asai
  3. Yizhong Wang
  4. Sean Welleck
  5. Hong-Liu Cao
  6. Stella Biderman
  7. Albert Qiaochu Jiang
  8. Zhangir Azerbayev

In Retrieval-Augmented Generation

  1. Hannaneh Hajishirzi
  2. Akari Asai
  3. Yizhong Wang
  4. Hong-Liu Cao

In Large Language Models

  1. Hailey Schoelkopf
  2. Sean Welleck
  3. Stella Biderman
  4. Yizhong Wang
  5. Albert Qiaochu Jiang
  6. Zhangir Azerbayev
  7. Hong-Liu Cao
  8. Marco Dos Santos

Citations in the corpus