Complementing Lexical Retrieval with Semantic Residual Embedding

By Luyu Gao, Zhuyun Dai, Zhenhua Fan, Jamie Callan. 2020

91 citations recorded by the source corpus.

Information retrieval traditionally has relied on lexical matching signals, but lexical matching cannot handle vocabulary mismatch or topic-level matching. Neural embedding based retrieval models can match queries and documents in a latent semantic space, but they lose token-level matching information that is critical to IR. This paper presents CLEAR, a deep retrieval model that seeks to complement lexical retrieval with semantic embedding retrieval. Importantly, CLEAR uses a residual-based embedding learning framework, which focuses the embedding on the deep language structures and semantics that the lexical retrieval fails to capture. Empirical evaluation demonstrates the advantages of CLEAR over classic bag-of-words retrieval models, recent BERT-enhanced lexical retrieval models, as well as a BERT-based embedding retrieval. A full-collection retrieval with CLEAR can be as effective as a BERT-based reranking system, substantially narrowing the gap between full-collection retrieval and cost-prohibitive reranking systems

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “All DR baselines use the same BERT-Siamese (base) model as used in various parallel research [9, 12, 13, 15].”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “While there are aspects of text ranking with pretrained trans-formers that are well understood, many promising directions await further exploration.”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, neutral): “Prior and concurrent work has also used hybrid sparse-dense models (Guo et al., 2016a; Seo et al., 2019; Karpukhin et al., 2020; Ma et al., 2020; Gao et al., 2020).”
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval (2021, background): “Well-known examples include DPR (Karpukhin et al., 2020), ANCE (Xiong et al., 2021), and ColBERT (Khattab and Zaharia, 2020), but there is much recent work along these lines (Gao et al., 2020; Hofstätter et al., 2020; Hof-stätter et al., 2021; Lin et al., 2021b), just to list a few papers.”
Optimizing Dense Retrieval Model Training with Hard Negatives (2021, methodology): “We use the open-sourced BM25 Neg model and ANCE model [29].”
Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance (2021, background): “BM25 [40] is a popular probabilistic BoW retrieval model that ranks documents based on the query terms appearing in each document.”
Answer Complex Questions: Path Ranker Is All You Need (2021, background): “However, the two-tower dense retriever possesses weak interactions between query and documents [3, 9, 14] , and suffers from the irrelevance of questions and documents.”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, methodology): “A cross-encoder has been shown to be an effective teacher (Hofstätter et al., 2020; Gao et al., 2020a) since it allows rich interactions between the intermediate transformer representations of a query q and a passage p.”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, background): “This dense approach was recently extended by hybrid lexical-dense approaches which aims to combine the strengths of both approaches (Gao et al., 2020; Seo et al., 2019; Luan et al., 2021).”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, unknown): “These deep LMs have been exploited to improve search along four main patterns, which are characterised by a trade-off between effectiveness and efficiency, exposed in the high-level comparison of • representation based methods , like EPIC [16], ColBERT [12], RepBERT [31], CLEAR [10] and ANCE [27].”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, unknown): “…for dense retrieval, exempliﬁed by DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2021), but also encompassing many other techniques (Gao et al., 2021b; Hofstätter et al., 2020; Qu et al., 2021; Hofstätter et al., 2021; Lin Table 1: Our conceptual framework for organizing recent…”
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers (2020, methodology): “When we further incorporate sparse signals, our proposed method beats the current state of the art in hybrid approaches, CLEAR (Gao et al., 2020), in both the MS MARCO and TREC 2019 DL tasks.”