Autoregressive Search Engines: Generating Substrings as Document Identifiers

By Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen-tau Yih, Sebastian Riedel, Fabio Petroni. 2022

222 citations recorded by the source corpus.

Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus. Autoregressive language models are emerging as the de-facto standard for generating answers, with newer and more powerful systems emerging at an astonishing pace. In this paper we argue that all this (and future) progress can be directly applied to the retrieval problem with minimal intervention to the models' architecture. Previous work has explored ways to partition the search space into hierarchical structures and retrieve documents by autoregressively generating their unique identifier. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers. This setup allows us to use an autoregressive model to generate and score distinctive ngrams, that are then mapped to full passages through an efficient data structure. Empirically, we show this not only outperforms prior autoregressive approaches but also leads to an average improvement of at least 10 points over more established retrieval solutions for passage-level retrieval on the KILT benchmark, establishing new state-of-the-art downstream performance on some datasets, while using a considerably lighter memory footprint than competing systems. Code and pre-trained models at https://github.com/facebookresearch/SEAL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Re2G: Retrieve, Rerank, Generate (2022, background): “SEAL [Bevilacqua et al., 2022] The novel tells the story of Dracula’s attempt to move from Transylvania to England so that he may ﬁnd new blood and spread the undead curse, and of the battle between Dracula and a small group of men and a woman led by Professor Abraham Van Helsing.”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “Generative Retrieval Generative search is a new class of retrieval methods that use neural generative models as search indices (Metzler et al., 2021; Tay et al., 2022; Bevilacqua et al., 2022; Lee et al., 2022).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Later extensions almost follow the three major methods for representing docids: DynamicRetriever [301] (atomic docid: original docid), SEAL [302] (simple string docid: full ngram signatures with the efficient support of Ferragina Manzini index), NCI [303] (semantic string docids: hierarchical cluste”
Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “However, one needs to create the identiﬁers, hence the structure was not thoroughly evaluated on a large-scale benchmark (Bevilacqua et al., 2022).”
Large Language Models for Information Retrieval: A Survey (2023, unknown): “Given the abundance of documents, a straightforward idea is to use LLMs for generating their corresponding pseudo queries.”
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “Unlike DSI, which unifies the index into the model parameters, SEAL requires a separate index data structure to perform an efficient search.”
Retrieval And Structuring Augmented Generation with Large Language Models (2025, unknown): “Recently, Generative Retrieval (GR) [13, 41, 231, 236, 252] has emerged as a new paradigm leveraging generative model [2, 206, 239] parameters for memorizing documents, enabling retrieval by directly generating relevant document identifiers.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “Followup works (Wang et al., 2022; Bevilacqua et al., 2022) further improve upon this paradigm with strategies to construct semantic DocIDs and enable robust training.”