Sparse, Dense, and Attentional Representations for Text Retrieval

By Yi Luan, Jacob Eisenstein, Kristina Toutanova, Michael Collins. 2020

509 citations recorded by the source corpus.

Abstract Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “Finally, Luan et al. (2020) conducts a theoretical and empirical study of sparse, dense and cross-attention information retrieval systems.”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “All DR baselines use the same BERT-Siamese (base) model as used in various parallel research [9, 12, 13, 15].”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, neutral): “retrieval results (Luan et al., 2020) (despite sacrificing search efficiency) and using more paragraphs increases the recall of matched answer spans3, this concurrent result is not directly comparable to ours, and we leave the combination effect of efficient pre-”
Improving Wikipedia verifiability with AI (2022, methodology): “The dense retrieval sub-system is a neural network which learns from Wikipedia data to encode the citation context into a dense query vector (Wu et al., 2019; Karpukhin et al., 2020; Maillard et al., 2021; Oğuz et al., 2021; Luan et al., 2021).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “It was suggested before that bi-encoders may be inherently not expressive enough for the purpose of large scale retrieval (Luan et al., 2021).”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, neutral): “Luan et al. (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross-and poly-encoders.”
You Only Need One Model for Open-domain Question Answering (2021, methodology): “Per-token embeddings or multiple embeddings were used in ColBERT (Khattab et al., 2020), ME-BERT (Luan et al., 2021), Lin et al. (2021), Lee et al. (2021).”
Learning to Retrieve Passages without Supervision (2021, methodology): “A similar approach was considered by Luan et al. (2021).”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “The progress can be attributed to advances in neural retrieval methods (Karpukhin et al., 2020; Izacard and Grave, 2020; Khattab et al., 2020; Luan et al., 2021; Xiong et al., 2020, inter alia) that benefit from smarter negative sampling strategies or a better trade-off between complex…”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, background): “There are two broad classes of neural retrievers based on the number of embeddings computed for a document: dual encoders (Yih et al., 2011, Lee et al., 2019) and multivector encoders (Khattab and Zaharia, 2020, Luan et al., 2021).”
A Memory Efficient Baseline for Open Domain Question Answering (2020, methodology): “This follows the same solution adopted in (Luan et al., 2020), where the trade-off between the dimension reduction and the retrieval performance is studied.”
Generation-Augmented Retrieval for Open-Domain Question Answering (2020, background): “However, the dense models are more computationally expensive and suffer from information loss as they condense the entire text sequence into a ﬁxed-size vector that does not guarantee exact matching (Luan et al., 2020).”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “While, hard negatives are shown to be important to train a dual-encoder (Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020; Xiong et al., 2020).”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The irregular representation space raises the risk of degeneration [37, 55] and often necessitates sophisticated post-adjustment or ﬁne-tuning to improve the sequence representations [16, 30, 32, 60].”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “[35] represent each passage with a fixed number of contextualized embeddings of the CLS token and the firstm token of the passage and score the relevance of the passage with the maximum score of the embeddings.”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, methodology): “Many papers (Luan et al., 2021; Gao et al., 2020b; Ma et al., 2021; Lin et al., 2021) have demonstrated that sparse retrieval can complement dense retrieval via a simple linear combination of their scores.”
SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval (2020, background): “Forthe first-stage retrieval, neural dense representations show great potentials for semantic matching and outperform sparse meth-ods in many NLP tasks, but this is not necessarily true in scenarios that emphasize long document retrieval and exact matching[9].”