Learning to Match using Local and Distributed Representations of Text for Web Search

By Bhaskar Mitra, Fernando Diaz, Nick Craswell. 2016

512 citations recorded by the source corpus.

Models such as latent semantic analysis and those based on neural embeddings learn distributed representations of text, and match the query against the document in the latent semantic space. In traditional information retrieval models, on the other hand, terms have discrete or local representations, and the relevance of a document is determined by the exact matches of query terms in the body text. We hypothesize that matching with distributed representations complements matching with traditional local representations, and that a combination of the two is favourable. We propose a novel document ranking model composed of two separate deep neural networks, one that matches the query and the document using a local representation, and another that matches the query and the document using learned distributed representations. The two networks are jointly trained as part of a single neural network. We show that this combination or 'duet' performs significantly better than either neural network individually on a Web page ranking task, and significantly outperforms traditional baselines and other recently proposed models based on neural networks.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Representation & Self-Supervised Learning

Citations in the corpus

End-to-End Neural Ad-hoc Ranking with Kernel Pooling (2017, background): “Interaction based models and representation based models address the ranking task from dierent perspectives, and can be combined for further improvements [18].”
Passage Re-ranking with BERT (2019, methodology): “The information retrieval (IR) community has also experienced a ﬂourishing development of neural ranking models, such as DRMM (Guo et al., 2016), KNRM (Xiong et al., 2017), Co-PACRR (Hui et al., 2018), and DUET (Mitra et al., 2017).”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “In this work, we compare against KNRM [5, 36], Duet [21, 23], ConvKNRM [5], and fastText+ConvKNRM [11].”
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale (2019, background): “Information Retrieval Success in deep neural networks inspires their application to information retrieval (IR) tasks (Huang et al., 2013; Guo et al., 2016; Mitra et al., 2017; Dehghani et al., 2017).”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, methodology): “, 2016), DUET (Mitra et al., 2017), KNRM (Xiong et al.”
Combining Fact Extraction and Verification with Neural Semantic Matching Networks (2018, background): “Information Retrieval: Recent success in deep neural networks has brought increasing interest in their application to information retrieval (IR) tasks (Huang et al. 2013; Guo et al. 2016; Mitra, Diaz, and Craswell 2017; Dehghani et al. 2017).”
Multi-Stage Document Ranking with BERT (2019, methodology): “Well-known neural ranking models include DRMM (Guo et al., 2016), DUET (Mitra et al., 2017), KNRM (Xiong et al., 2017), and Co-PACRR (Hui et al., 2018); the literature is too vast for an exhaustive review here, and thus we refer readers to recent overviews (Onal et al., 2018; Mitra and Craswell,…”
Simple Applications of BERT for Ad Hoc Document Retrieval (2019, neutral): “Note that for these comparisons we leave aside many non-neural approaches that take advantage of learning-to-ranking techniques over manually-2011 (Guo et al., 2016) 0.3477 0.4034 0.2213 0.3537 0.2639 0.4772 0.4042 0.6139 DUET (Mitra et al., 2017) 0.3576 0.4000 0.2243 0.3644 0.2779 0.4878 0.4219 0.6”
Neural Ranking Models with Weak Supervision (2017, background): “[28] proposed to simultaneously learn local and distributional representations, which are early and late combination models respectively, to capture both exact termmatching and semantic termmatching.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “, 2017), and DUET (Mitra et al., 2017), have been proposed to re-rank the top-k candidates.”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, neutral): “Neural ranking models have shown great potential and have been widely adopted in the IR community (Dehghani et al., 2017; Guo et al., 2019; Mitra et al., 2017; Zamani et al., 2018, inter-alia).”
Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search (2018, unknown): “e interaction based model and the representation based model can also be combined in a duet architecture [21].”
BISON: BM25-weighted Self-Attention Framework for Multi-Fields Document Search (2020, unknown): “Similar with (18; 20; 11) we sample 30 million query-document pairs from Bing’s search log.”
Co-PACRR: A Context-Aware Neural IR Model for Ad-hoc Retrieval (2017, methodology): “cument pairs, denoted as PairAccuracy. We compare our model with multiple state-of-theart neural IR models including the PACRR model [10], MatchPyramid [18], DRMM [7], the local model of DUET (DUETL) [15], and the most recent K-NRM [23] model. As discussed in Section 2, our focus is on evaluating de”
Ranking of text documents using TF-IDF weighting and association rules mining (2018, background): “Authors in this paper [13] propose a document ranking model composed of two separate deep neural networks, one that matches the query and the document using a local representation, and another that matches the query and the document using learned distributed representations.”
An anatomy for neural search engines (2019, methodology): “• There was no retrieval phase, the model was tested directly against a labeled set of documents, which is not realistic in a real-world application [Huang et al., 2013; Shen et al., 2014b,a; Palangi et al., 2016; Zamani et al., 2018; Pang et al., 2017b; Xiong et al., 2017a; Mitra et al., 2017].”
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks (2019, methodology): “In this preliminary study, we incorporate the query term independence assumption into three state-of-the-art neural ranking models—BERT [Nogueira and Cho, 2019], Duet [Mitra et al., 2017], and CKNRM [Dai et al.”
Rethinking Search: Making Experts out of Dilettantes (2021, methodology): “models include Deep Relevance Matching Model (DRMM) [32], DUET [60], Kernel-based Neural Ranking Model (KNRM) [101], Position-Aware Convolutional-Recurrent Relevance (PACRR) [34], and Context-Aware PACRR (co-PACRR) [35].”
MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval (2020, methodology): “g: in [7], authors proposed a duet architecture composed of two deep neural networks, a local model that captures exact matching signals and a distributed model for semantic matching.”