Transformer Memory as a Differentiable Search Index

By Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta and 3 more. 2022

444 citations recorded by the source corpus.

In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yi Tay
  6. Hannaneh Hajishirzi
  7. Graham Neubig
  8. Yejin Choi

In Computer Networks and Communications

  1. Denny Zhou
  2. Shashank Rajput
  3. Wen-tau Yih
  4. Dan Roth
  5. Meng Jiang
  6. Patrick Lewis
  7. Fernando Diaz
  8. Jian Pei

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Andrew Zisserman
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Cordelia Schmid
  8. Ming-Hsuan Yang

In Retrieval-Augmented Generation

  1. Sebastian Riedel
  2. Fabio Petroni
  3. Wen-tau Yih
  4. Luke S. Zettlemoyer
  5. Chenguang Zhu
  6. Meng Jiang
  7. W. Yu
  8. Armen Aghajanyan

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Denny Zhou
  3. Yi Tay
  4. Sebastian Riedel
  5. Fabio Petroni
  6. Xuezhi Wang
  7. Jason Wei
  8. Ed H. Chi

Citations in the corpus