End-to-End Neural Ad-hoc Ranking with Kernel Pooling

By Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, Russell Power. 2017

591 citations recorded by the source corpus.

This paper proposes K-NRM, a kernel based neural model for document ranking. Given a query and a set of documents, K-NRM uses a translation matrix that models word-level similarities via word embeddings, a new kernel-pooling technique that uses kernels to extract multi-level soft match features, and a learning-to-rank layer that combines those features into the final ranking score. The whole model is trained end-to-end. The ranking layer learns desired feature patterns from the pairwise ranking loss. The kernels transfer the feature patterns into soft-match targets at each similarity level and enforce them on the translation matrix. The word embeddings are tuned accordingly so that they can produce the desired soft matches. Experiments on a commercial search engine's query log demonstrate the improvements of K-NRM over prior feature-based and neural-based states-of-the-art, and explain the source of K-NRM's advantage: Its kernel-guided embedding encodes a similarity metric tailored for matching query words to document words, and provides effective multi-level soft matches.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “We use a pretrained XLNet to extract word embeddings for the documents and queries without ﬁnetuning, and employ a kernel pooling network [36] to rank the documents.”
Word-Entity Duet Representations for Document Ranking (2017, methodology): “More sophisticated neural ranking models [22] can also be applied with the word-entity duet, especially when more training data are available.”
Passage Re-ranking with BERT (2019, methodology): “The information retrieval (IR) community has also experienced a ﬂourishing development of neural ranking models, such as DRMM (Guo et al., 2016), KNRM (Xiong et al., 2017), Co-PACRR (Hui et al., 2018), and DUET (Mitra et al., 2017).”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “This overthrows a previously-held belief that it is necessary to capture the interactions between the discrete query and document terms [34, 35].”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, methodology): “Instead of summarizing q and d into individual embeddings, these rankers model word-and phrase-level relationships across q and d and match them using a deep neural network (e.g., with CNNs/MLPs [23] or kernels [36]).”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, background): “, 2017), KNRM (Xiong et al., 2017), and CoPACRR (Hui et al.”
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval (2019, methodology): “It is also more accurate than a popular neural re-ranking model K-NRM [33].”
Simple Applications of BERT for Ad Hoc Document Retrieval (2019, neutral): “…0.3477 0.4034 0.2213 0.3537 0.2639 0.4772 0.4042 0.6139 DUET (Mitra et al., 2017) 0.3576 0.4000 0.2243 0.3644 0.2779 0.4878 0.4219 0.6467 K-NRM (Xiong et al., 2017) 0.3576 0.4000 0.2277 0.3520 0.2721 0.4756 0.4137 0.6358 PACRR (Hui et al., 2017) 0.3810 0.4286 0.2311 0.3576 0.2803 0.4944 0.4140…”
Understanding the Behaviors of BERT in Ranking (2019, methodology): “Different from seq2seq learning, previous Neu-IR research considers such surroundingcontext-trained neural models not as effective in search as relevance modeling [7, 8].”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “, 2016), KNRM (Xiong et al., 2017), and DUET (Mitra et al.”
Context-Aware Document Term Weighting for Ad-Hoc Search (2020, background): “Most of prior work uses dense text representations that enable a query to match every document to some degree, which makes them impractical for first-stage ranking in large-scale datasets [11, 13, 16, 18, 39].”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, methodology): “tors are learned functions from training data. Interaction models, on the other hand, learn patterns indicative of relevance. For instance, PACRR [19] learns soft n-gram matches in the text, and KNRM [44] learns matching kernels based on word similarity scores between the query and the document. 2.2”
Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension (2018, background): “al distracting sentences [23]. Neural IR. Neural ranking models are currently a populartopic in IR [30]. There are roughly two groups: embedding space models [19, 32, 41] and interaction-based models [10, 16, 20, 21, 51]. In the embedding space models, the query and documents are ﬁrst embedded into ”
Training Curricula for Open Domain Answer Re-Ranking (2020, background): “odels the term occurrences by building a histogram based on the similarity score for each query term and by producing a relevance score by feeding these histograms into a multi-layer perceptron. KNRM [37] works similarly, but replaces the hard histogram buckets with soft Gaussian-kernel-based bucket”
Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching (2020, background): “Note that we do not compare with any interaction-focused models like DRMM [8], K-NRM [33], Duet [20] or MatchPyramid [22].”
Rethinking Search: Making Experts out of Dilettantes (2021, methodology): “Some examples of such models include Deep Relevance Matching Model (DRMM) [32], DUET [60], Kernel-based Neural Ranking Model (KNRM) [101], Position-Aware Convolutional-Recurrent Relevance (PACRR) [34], and Context-Aware PACRR (co-PACRR) [35].”