Relevance-based Word Embedding

By Hamed Zamani, W. Bruce Croft. 2017

167 citations recorded by the source corpus.

Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, has recently attracted much attention in natural language processing and information retrieval tasks. The embedding vectors are typically learned based on term proximity in a large corpus. This means that the objective in well-known word embedding algorithms, e.g., word2vec, is to accurately predict adjacent word(s) for a given word or context. However, this objective is not necessarily equivalent to the goal of many information retrieval (IR) tasks. The primary objective in various IR tasks is to capture relevance instead of term proximity, syntactic, or even semantic similarity. This is the motivation for developing unsupervised relevance-based word embedding models that learn word representations based on query-document relevance information. In this paper, we propose two learning models with different objective functions; one learns a relevance distribution over the vocabulary set for each query, and the other classifies each term as belonging to the relevant or non-relevant class for each query. To train our models, we used over six million unique queries and the top ranked documents retrieved in response to each query, which are assumed to be relevant to the query. We extrinsically evaluate our learned word representation models using two IR tasks: query expansion and query classification. Both query expansion experiments on four TREC collections and query classification experiments on the KDD Cup 2005 dataset suggest that the relevance-based word embedding models significantly outperform state-of-the-art proximity-based embedding models, such as word2vec and GloVe.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “An Action-Part Semantic Relevance-aware (APSR) framework is proposed to handle this task by utilizing the semantic relevance between each body part and each action class at the distributed word embedding level [27], [28].”
Understanding the Behaviors of BERT in Ranking (2019, methodology): “Different from seq2seq learning, previous Neu-IR research considers such surroundingcontext-trained neural models not as effective in search as relevance modeling [7, 8].”
Neural Ranking Models with Weak Supervision (2017, background): “Zamani and Croft [45] recently proposed relevance-based word embedding models for learning word representations based on the objectives that matter for IR applications. second category, which this paper belongs to, consists of the approaches that design and train a (deep) neural network for a…”
Context-Aware Document Term Weighting for Ad-Hoc Search (2020, methodology): “We compared HDCT, which uses discrete bag-of-words, to two retrieval models that use embeddings: RLM [41] and SNRM [42].”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, neutral): “Bag-of-words and contextual embedding models, such as word2vec and BERT, have also been explored extensively for various IR tasks, from document to sentence-level retrieval (Padigela et al., 2019; Zamani and Croft, 2016, 2017).”
Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search (2018, unknown): “Learned end-to-end from user feedbacks [23, 29], the word embeddings can encode so matches tailored for relevance ranking, which has signicant advantages over traditional feature-based methods [29, 30].”
Retrieval-Enhanced Machine Learning (2022, methodology): “Zamani and Croft [71] used the top retrieved documents to produce a relevance model distribution for training queries and learn relevance-based word embedding.”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (2018, unknown): “ACMReference Format: Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps.”
Co-PACRR: A Context-Aware Neural IR Model for Ad-hoc Retrieval (2017, background): “…relevance matching models, and hence the comparisons are limited to 1) modeling the interactions between a query and a document, excluding the learning-to-rank features for a single document or a query, e.g., PageRank scores, and 2) modeling relevance matching rather than semantic matching [7].”
A Deep Look into Neural Ranking Models for Information Retrieval (2019, methodology): “word embedding [111], and efficient learning to rank [112].”
To Clean or Not to Clean (2018, background): “These techniques are popular among researchers as baseline retrieval methods [37, 54, 73, 74, 78].”
Selective Weak Supervision for Neural Information Retrieval (2020, background): “Nevertheless, previous research finds that much of Neu-IR’s effectiveness relies on embeddings specifically trained for relevance, because the word co-occurrence trained word embeddings do not align well with the needs of ad hoc retrieval [45, 48].”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, background): “There has also been some effort in learning word embeddings to directly capture relevance matching [74, 21] rather than linguistic features as in word2vec [43] or GloVe [49].”
Word Embedding Causes Topic Shifting; Exploit Global Context! (2017, result): “More recently, the approaches to exploit the advancement in word embedding for IR has shown not only to be competitive to the local approaches but also that combining the approaches brings further improvements in comparison to each of them alone [12, 16, 17].”
Riker: Mining Rich Keyword Representations for Interpretable Product Question Answering (2019, methodology): “The mined keyword representations are employed as input to the IR module to alleviate its lexical gap problem and enhance the search performance, and are demonstrated more effective than the state-of-the-art word embedding based query expansion methods [12, 49].”