Transformer Memory as a Differentiable Search Index
By Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta and 3 more. 2022
444 citations recorded by the source corpus.
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.
Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yi Tay
- Hannaneh Hajishirzi
- Graham Neubig
- Yejin Choi
In Computer Networks and Communications
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Jitendra Malik
- Cordelia Schmid
- Ming-Hsuan Yang
In Retrieval-Augmented Generation
- Sebastian Riedel
- Fabio Petroni
- Wen-tau Yih
- Luke S. Zettlemoyer
- Chenguang Zhu
- Meng Jiang
- W. Yu
- Armen Aghajanyan
In Large Language Models
- Luke S. Zettlemoyer
- Denny Zhou
- Yi Tay
- Sebastian Riedel
- Fabio Petroni
- Xuezhi Wang
- Jason Wei
- Ed H. Chi
Citations in the corpus
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “DSI-BART On NQ320 k , bert-base-cased is used to compute the embeddings for the clustering.”
- UL2: Unifying Language Learning Paradigms (2022, background): “, differentiable search index (Tay et al., 2022) for our experiments.”
- Efficient Transformers: A Survey (2020, background): “Transformers are pervasive and have made tremendous impact in many fields such as language understanding [6, 12, 20, 60], image processing [7, 55] and information retrieval [77].”
- Retrieval-Enhanced Machine Learning (2022, background): “Tay et al. [57] proposed training a large model that memorizes the mapping of document content to document ids, which can be used to retrieve relevant document ids given a query at inference time.”
- Unifying Language Learning Paradigms (2022, background): “, differentiable search index (Tay et al., 2022) for our experiments.”
- Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “, on QA tasks) [19, 20, 21], while also allowing the models to encode significant amounts of world or factual knowledge [22, 23, 24].”
- Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “, 2020), root-to-leaf paths in a hierarchical cluster tree (Tay et al., 2022), or distinctive n-grams that can be mapped to full passages (Bevilacqua et al.”
- Recommender Systems with Generative Retrieval (2023, methodology): “, a single model can be used to train and serve without creating an index — the transformer memory acts as the index [34].”
- Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “Generative Retrieval Generative search is a new class of retrieval methods that use neural generative models as search indices (Metzler et al., 2021; Tay et al., 2022; Bevilacqua et al., 2022; Lee et al., 2022).”
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “For example, SGPT (Muennighoff, 2022) generates text embeddings using GPT, generative document retrieval explores a differentiable search index (Tay et al., 2022; Cao et al., 2021; Sun et al., 2023), and HyDE (Gao et al., 2023; Wang et al., 2023a) generates pseudo-documents using GPT-3.”
- Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “For training the original DSI model, we use the training configuration suggested in the original paper [43].”
- LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, background): “For example, generative retrieval [107–109] is a recent class of models performing retrieval by directly predicting the document IDs, which could be the premise names in our task.”
- Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Further, a representative work [298] called DSI develops a generative retrieval model based on a seq2seq encoder-decoder architecture (i.e., T5), where query and docids correspond to input and output, respectively.”
- Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, methodology): “Furthermore, the comparisons between different methods using the same LLM are fair - PRP always outperforms pointwise baselines by a large margin, and listwise prompting almost always fails on moderate LLMs. Avoiding data leakage in the era of LLM is generally challenging and more rigorous…”
- Large Language Models for Information Retrieval: A Survey (2023, unknown): “A common practice to improve the models’ effectiveness in a target domain without adequate label signals is through data augmentation.”
- ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “The first class of systems is vanilla LLM prompting (Tay et al., 2022; Weller et al., 2023), where LLMs such as GPT-4 (OpenAI, 2023) are prompted to return attributions (in the form of titles of references, optionally accompanied with URLs) along with their answers.”
- Retrieval And Structuring Augmented Generation with Large Language Models (2025, unknown): “Recently, Generative Retrieval (GR) [13, 41, 231, 236, 252] has emerged as a new paradigm leveraging generative model [2, 206, 239] parameters for memorizing documents, enabling retrieval by directly generating relevant document identifiers.”
- A Survey of Model Architectures in Information Retrieval (2025, unknown): “In the decoding phase, DSI uses all DocIDs to construct a trie and performs constrained decoding.”