Autoregressive Entity Retrieval

By Nicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio Petroni. 2020

551 citations recorded by the source corpus.

Entities are at the center of how we represent and aggregate knowledge. For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per article). The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. One way to understand current approaches is as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity representations produced by encoding entity information such as descriptions. This approach leads to several shortcomings: i) context and entity affinity is mainly captured through a vector dot product, potentially missing fine-grained interactions between the two; ii) a large memory footprint is needed to store dense representations when considering large entity sets; iii) an appropriately hard set of negative data has to be subsampled at training time. We propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion, and conditioned on the context. This enables to mitigate the aforementioned technical issues: i) the autoregressive formulation allows us to directly capture relations between context and entity name, effectively cross encoding both; ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; iii) the exact softmax loss can be efficiently computed without the need to subsample negative data. We show the efficacy of the approach with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new SOTA, or very competitive results while using a tiny fraction of the memory of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their unambiguous name.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, neutral): “Model AIDA FEV T-REx zsRE NQ HoPo TQA WoW acc acc acc acc em em em f1 GENRE (Cao et al., 2021) 89.”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “GENRE [Cao et al., 2021] is trained on BLINK [Wu et al., 2020] and all KILT tasks jointly using a sequence-to-sequence language model to generate the title of the Wikipedia page where the answer can be found.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “On NQ320k, a page-level benchmark, we include our own replication of GENRE (De Cao et al., 2021b).”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “We evaluate on the AIDA-CoNLL dataset for the in-domain scenario and the MSNBC, AQUAINT, ACE2004, WNED-CWEB (CWEB) and WNED-WIKI (WIKI) for the out-of-domain scenario (De Cao et al., 2020; Guo & Barbosa, 2018).”
PEER: A Collaborative Language Model (2022, methodology): “For this variant, we additionally use constrained decoding as proposed by Cao et al. (2021) to ensure that the generated quote is actually contained in the cited document.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…architectures on text with entity coreference information yields more entity-aware LLMs, both in recurrent (Henaff et al., 2017; Ji et al., 2017; Yang et al., 2017; Liu et al., 2019) and Transformer-based models (Févry et al., 2020; De Cao et al., 2020; Rosset et al., 2020; Zhang et al., 2022c).”
Editing Factual Knowledge in Language Models (2021, background): “Notably, language models that only rely on internal implicit memory are state-of-the-art for (multilingual-) Entity Linking (De Cao et al., 2021a,b).”
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, neutral): “Exploring the trade-offs between storing and retrieving knowledge parametrically or non-parametrically is a topic of great current interest (Lewis et al., 2020a; De Cao et al., 2020), and PAQ should be a useful testbed for probing this relationship further.”
Multi-Task Retrieval for Knowledge-Intensive Tasks (2021, background): “Entity linking is a task that is normally better performed by models which are explicitly designed for it (Cao et al., 2020).”
Transformer Memory as a Differentiable Search Index (2022, background): “De Cao et al. (2020) describe a related sequence-to-sequence system called autoregressive entity linking , in which documents mentioning an entity—perhaps implicitly, e.g., by posing a question to which that entity is an answer—are mapped to a canonical name of that entity.”
Robust Retrieval Augmented Generation for Zero-shot Slot Filling (2021, methodology): “To this aim, we adopt the technique described in (De Cao et al., 2020) to restrict the vocabulary of tokens during the generation.”
Relational Memory-Augmented Language Models (2022, background): “, 2018), surface forms of entities (Kiddon et al., 2016; Yang et al., 2017; Cao et al., 2021), entity types (Parvez et al.”
Answer Generation for Retrieval-based Question Answering Systems (2021, background): “Recent works have shown that large, transformerbased conditional generative models can be used to significantly improve parsing (Chen et al., 2020; Rongali et al., 2020), retrieval (De Cao et al., 2020; Pradeep et al., 2021), and classification tasks (Raffel et al., 2019).”
A Unified Encoder-Decoder Framework with Entity Memory (2022, methodology): “Readers can refer to (Cao et al., 2021) for more implementation details.”
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “For example, SGPT (Muennighoff, 2022) generates text embeddings using GPT, generative document retrieval explores a differentiable search index (Tay et al., 2022; Cao et al., 2021; Sun et al., 2023), and HyDE (Gao et al., 2023; Wang et al., 2023a) generates pseudo-documents using GPT-3.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “Indeed, the idea of generative retrieval has been first explored in the task of entity linking [299], where entity identifiers (e.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “To improve the model, a trie-based constrained decoding procedure [15, 30] is introduced to ensure the generation of valid event structures.”
Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction (2021, methodology): “To exploit the event schema knowledge, we pro-pose to employ a trie-based constrained decoding algorithm (Chen et al., 2020a; Cao et al., 2021) for event generation.”
Recommender Systems with Generative Retrieval (2023, methodology): “[5] proposed GENRE for entity retrieval, which used a transformer-based architecture to return, token-by-token, the name of the entity referenced to in a given query.”