Learning Dense Representations for Entity Retrieval

By D. Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, Diego Garcia-Olano. 2019

230 citations recorded by the source corpus.

We show that it is feasible to perform entity linking by training a dual encoder (two-tower) model that encodes mentions and entities in the same dense vector space, where candidate entities are retrieved by approximate nearest neighbor search. Unlike prior work, this setup does not rely on an alias table followed by a re-ranker, and is thus the first fully learned entity retrieval model. We show that our dual encoder, trained using only anchor-text links in Wikipedia, outperforms discrete alias table and BM25 baselines, and is competitive with the best comparable results on the standard TACKBP-2010 dataset. In addition, it can retrieve candidates extremely fast, and generalizes well to a new dataset derived from Wikinews. On the modeling side, we demonstrate the dramatic value of an unsupervised negative mining algorithm for this task.

Subfields: Artificial Intelligence, Information Retrieval & Search, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Representation & Self-Supervised Learning

Citations in the corpus

Sparse, Dense, and Attentional Representations for Text Retrieval (2020, background): “A promising alternative first-stage retriever is one based on learned dense low-dimensional encodings of documents and queries (Huang et al., 2013; Reimers and Gurevych, 2019; Gillick et al., 2019; Karpukhin et al., 2020).”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “In NLP, a similar contrastive learning objective has been explored in different contexts (Henderson et al., 2017; Gillick et al., 2019; Karpukhin et al., 2020).”
Zero-shot Entity Linking with Dense Entity Retrieval (2019, methodology): “Following prior work (Sun et al., 2015; Cao et al., 2018; Gillick et al., 2019), we pre-train our models on Wikipedia 4 data.”
Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2019, methodology): “Following prior work (Sun et al., 2015; Cao et al., 2018; Gillick et al., 2019; Onoe and Durrett, 2019), we pre-train our models on Wikipedia6 data.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “While, hard negatives are shown to be important to train a dual-encoder (Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020; Xiong et al., 2020).”
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, methodology): “…with randomly initialized embeddings that are updated during training, using between 1 and 50 in-batch negative samples, which is a standard method in the knowledge base embedding literature (Bordes et al., 2013) and has been used successfully for entity retrieval (Gillick et al., 2019).”
One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (2021, methodology): “For each question, we use positive passages for the other questions in the training batch as negative passages ( in-batch negative , Gillick et al., 2019; Karpukhin et al., 2020). mDPR is updated by minimizing the negative log likelihood of positive passages: .”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “Following Henderson et al. (2017); Gillick et al. (2019); Karpukhin et al. (2020), we use in-batch negatives to avoid the actual sampling of a negative image or text: given a batch of n positive image-text pairs B = { ( i 1 , t 1 ) , . . . , ( i n , t n ) } , we use all other images from within the…”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “Such a two-stage procedure is particularly useful in a variety of natural language processing tasks, including question answering (Mao et al., 2021; Xiong et al., 2020b), dialogue system (Ji et al., 2014; Henderson et al., 2017) and entity linking (Gillick et al., 2019; Wu et al., 2020).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “As one of the most essential techniques to overcome information overload, text retrieval systems have been widely employed to support many downstream applications, including question answering [1], [2], dialog system [3], [4], entity linking [5], [6] and Web search [7].”
RepBERT: Contextualized Text Embeddings for First-Stage Retrieval (2020, methodology): “Such trick has been used in prior works [12, 17] for training a siamese neural network.”
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval (2021, background): “…distance ( i.e., dissimilarity) for visualization: the longer the distance is, the less similar it is. et al., 2020b), information retrieval (Luan et al., 2021; Khattab and Zaharia, 2020), dialogue (Ji et al., 2014; Henderson et al., 2017) and entity linking (Gillick et al., 2019; Wu et al., 2020).”
Accelerating Large-Scale Inference with Anisotropic Vector Quantization (2019, methodology): “In this framework, we learn embedding functions to map items to be retrieved in a common vector space, where the items can be words, images, users, audio, products, web pages, graph nodes, or anything of interest (Cremonesi et al., 2010; Weston et al., 2010; Guo et al., 2016a; Gillick et al., 2019; ”