End-to-End Retrieval in Continuous Space

By D. Gillick, Alessandro Presta, Gaurav Singh Tomar. 2018

108 citations recorded by the source corpus.

Most text-based information retrieval (IR) systems index objects by words or phrases. These discrete systems have been augmented by models that use embeddings to measure similarity in continuous space. But continuous-space models are typically used just to re-rank the top candidates. We consider the problem of end-to-end continuous retrieval, where standard approximate nearest neighbor (ANN) search replaces the usual discrete inverted index, and rely entirely on distances between learned embeddings. By training simple models specifically for retrieval, with an appropriate model architecture, we improve on a discrete baseline by 8% and 26% (MAP) on two similar-question retrieval tasks. We also discuss the problem of evaluation for retrieval systems, and show how to modify existing pairwise similarity datasets for this purpose.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “In the context of ad-hoc document retrieval, Gillick et al. (2018) showed that bi-encoder models could be competitive with traditional IR systems. For open domain question-answering, Karpukhin et al. (2020) introduced dense passage retrieval (DPR), which uses dense embeddings and nearest neighbors s”
Learning Dense Representations for Entity Retrieval (2019, positive): “To estimate performance in a real-world setting, we repeat the evaluation of DEER using the quantization-based approaches described by Guo et al. (2016).”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Gillick et al. (2018) ﬁrst studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking.”
Fusion of Detected Objects in Text for Visual Question Answering (2019, background): “Dual Encoders, discussed for example by Wu et al. (2018) and Gillick et al. (2018), are models that embed objects of potentially different types into a common representation space where a similarity function can be expressed e.g. as a dot product or a cosine similarity.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “We train an image-text dual encoder (Figure 5) [11, 5, 40] for cross-modal retrieval [9], using the model and code of Parekh et al.”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “We also consider several neural baseline models, all of which are dual encoders (Gillick et al., 2018; Yang et al., 2019) that encode both inputs separately.”
Transformer Memory as a Differentiable Search Index (2022, background): “For retrieval, approaches based on inverted indexes or nearest neighbor search are common where contrastive learning based dual encoders (DEs) (Gillick et al., 2018; Karpukhin et al., 2020; Ni et al., 2021) are the present state-of-the-art.”
Approximate Nearest Neighbor Search and Lightweight Dense Vector Reranking in Multi-Stage Retrieval Architectures (2020, methodology): “In order to formulate a retrieval task (for example, in a community QA application), we followed exactly the procedure of Gillick et al. [8].”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, background): “Representation based models (Palangi et al., 2016; Gillick et al., 2018; Karpukhin et al., 2020) encode a query and passage independently into a common dense space, and scores their relevance based on vector dot-product or cosine similarity.”
Transferable Representation Learning in Vision-and-Language Navigation (2019, background): “The first is a two-tower model [15, 33] with a cross-modal alignment module.”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, methodology): “During fine-tuning, we include in-batch negatives (Gillick et al., 2018) together with hard-negatives as provided by MSMARCO.”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “For our baseline systems on both tasks, we follow previous work (Roy et al., 2020; Botha et al., 2020) and train a dual encoder initialized from the pre-trained model weights, optimizing for an inbatch sampled softmax loss (Gillick et al., 2018).”
Multilingual Universal Sentence Encoder for Semantic Retrieval (2019, methodology): “More details on the constructed datasets can be found in Gillick et al. (2018).”
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2021, methodology): “In contrast, sentence embeddings allow for efﬁcient retrieval and clustering (Gillick et al., 2018; Reimers and Gurevych, 2019; Yang et al., 2020).”
Sampling-bias-corrected neural modeling for large corpus item recommendations (2019, background): “Building neural networks with two towers has recently become a popular approach in several natural language tasks including modeling sentence similarities [31], response suggestions [24], and text-based information retrieval [17, 37].”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Gillick et al. (2018) ﬁrst studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking.”
Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation (2020, methodology): “Our retrieval model belongs to the family of Dual Encoders (DE) 2 (Gillick et al., 2018; Palangi et al., 2016) that encodes pairs of items in a shared space.”
ReQA: An Evaluation for End-to-End Answer Retrieval Models (2019, background): “There is growing interest in training end-to-end retrieval systems that can efﬁciently surface relevant results without an intermediate document retrieval phase (Gillick et al., 2018; Cakaloglu et al., 2018; Seo et al., 2019; Henderson et al., 2019).”
Large Dual Encoders Are Generalizable Retrievers (2021, background): “In the line of neural retrievers, dual encoders have demonstrated great performance compared to traditional sparse models, such as BM25, on a wide range of retrieval tasks (Karpukhin et al., 2020; Gillick et al., 2018).”