A Dual Embedding Space Model for Document Ranking

By Bhaskar Mitra, Eric T. Nalisnick, Nick Craswell, R. Caruana. 2016

140 citations recorded by the source corpus.

A fundamental goal of search engines is to identify, given a query, documents that have relevant text. This is intrinsically difficult because the query and the document may use different vocabulary, or the document may contain query words without being relevant. We investigate neural word embeddings as a source of evidence in document ranking. We train a word2vec embedding model on a large unlabelled query corpus, but in contrast to how the model is commonly used, we retain both the input and the output projections, allowing us to leverage both the embedding spaces to derive richer distributional relationships. During ranking we map the query words into the input space and the document words into the output space, and compute a query-document relevance score by aggregating the cosine similarities across all the query-document word pairs. We postulate that the proposed Dual Embedding Space Model (DESM) captures evidence on whether a document is about a query term in addition to what is modelled by traditional term-frequency based approaches. Our experiments show that the DESM can re-rank top documents returned by a commercial Web search engine, like Bing, better than a term-matching based signal like TF-IDF. However, when ranking a larger set of candidate documents, we find the embeddings-based approach is prone to false positives, retrieving documents that are only loosely related to the query. We demonstrate that this problem can be solved effectively by ranking based on a linear mixture of the DESM and the word counting features.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “, 2014], and the CK model uses a concatenation of word2vec’s “IN” and “OUT” embeddings [Mitra et al., 2016].”
Learning to Match using Local and Distributed Representations of Text for Web Search (2016, methodology): “Among the baseline models, including both traditional and neural network based models, CDSSM and DESM achieve the highest NDCG at position one and ten, respectively, on the weighted test set.”
Improving Document Ranking with Dual Word Embeddings (2016, methodology): “We provide additional analysis and experiment results in [4].”
Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “We use a concatenation of the IN and OUT embeddings [Nalisnick et al., 2016, Mitra et al., 2016] from word2vec to initialize the embedding layer parameters.”
Neural information retrieval: at the end of the early years (2017, methodology): “See Mitra et al [145] for further discussion of such two-stage telescoping approaches.”
PACRR: A Position-Aware Neural IR Model for Relevance Matching (2017, background): “Examples in this regard include DSSM (Huang et al., 2013) and DESM (Mitra et al., 2016).”
Ranking of text documents using TF-IDF weighting and association rules mining (2018, background): “Authors in this paper [9] investigate neural word embeddings as a source of evidence in document ranking.”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, background): “…cross-lingual tasks such as bilingual lexicon induction (Mikolov et al., 2013a; Gouws et al., 2015; Duong et al., 2016), cross-lingual information retrieval (Vuli´c and Moens, 2015; Mitra et al., 2016), and transfer learning for resource-lean languages (Sø-gaard et al., 2015; Guo et al., 2015).”
Efficient Interaction-based Neural Ranking with Locality Sensitive Hashing (2019, background): “firstly introduced Dual Embedding [21] for document ranking, given the fact that word2vec [19] with Negative Sampling actually trains two sets of word embedding vectors, defined as “input” (IN) and “output” (OUT) vector representations.”
EXS: Explainable Search Using Local Model Agnostic Interpretability (2018, unknown): “…networks have been widely successful in an array of learning tasks including vision, speech and NLP. Recently, the IR community has also seen successful application of deep learning models in document representation [3, 9], query modeling [4, 19, 20] and design of ranking models [6, 10, 11, 16].”
Co-PACRR: A Context-Aware Neural IR Model for Ad-hoc Retrieval (2017, background): “…relevance matching models, and hence the comparisons are limited to 1) modeling the interactions between a query and a document, excluding the learning-to-rank features for a single document or a query, e.g., PageRank scores, and 2) modeling relevance matching rather than semantic matching [7].”