Supervised semantic indexing

By Bing Bai, Jason Aaron Edward Weston, David Grangier, Ronan Collobert, Kunihiko Sadamasa, Yanjun Qi, Olivier Chapelle, Kilian Q. Weinberger. 2009

107 citations recorded by the source corpus.

In this article we propose Supervised Semantic Indexing (SSI), an algorithm that is trained on (query, document) pairs of text documents to predict the quality of their match. Like Latent Semantic Indexing (LSI), our models take account of correlations between words (synonymy, polysemy). However, unlike LSI our models are trained with a supervised signal directly on the ranking task of interest, which we argue is the reason for our superior results. As the query and target texts are modeled separately, our approach is easily generalized to different retrieval tasks, such as online advertising placement. Dealing with models on all pairs of words features is computationally challenging. We propose several improvements to our basic model for addressing this issue, including low rank (but diagonal preserving) representations, and correlated feature hashing (CFH). We provide an empirical study of all these methods on retrieval tasks based on Wikipedia documents as well as an Internet advertisement task. We obtain state-of-the-art performance while providing realistically scalable methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (2019, methodology): “Such methods include vector space models (Salton et al., 1975), LSI (Deerwester et al., 1990), supervised embeddings (Bai et al., 2009; Wu et al., 2018) and classical siamese networks (Bromley et al., 1994).”
Learning End-to-End Goal-Oriented Dialog (2016, background): “Firstly, as words are embedded into a low dimensional space the model is often unable to differentiate between exact word matches, and matches between words with similar meaning (Bai et al. , 2009).”
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems (2015, methodology): “We evaluate on these tasks the performance of various neural network models that can potentially create end-to-end dialogs, ranging from simple supervised embedding models (Bai et al., 2009), RNNs with Long Short-Term Memory (LSTMs) (Hochreiter & Schmidhuber, 1997), and attention-based models, in pa”
StarSpace: Embed All The Things! (2017, methodology): “r unsupervised embeddings; we also compare on the SentEval tasks (Conneau et al. 2017) against a wide range of unsupervised models for sentence embedding. In the domain of supervised embeddings, SSI (Bai et al. 2009) and WSABIE (Weston, Bengio, and Usunier 2011) areearly approachesthat showedpromise”
Aggregating Continuous Word Embeddings for Information Retrieval (2013, methodology): “Note that we did not compute topic models with LDA on these datasets as LSI provides similar performances to LDA (Wang et al., 2011; Bai et al., 2009).”
Semantic Matching in Search (2014, background): “[7, 8], takes the same view on the problem.”
Neural information retrieval: at the end of the early years (2017, methodology): “For learning parameters of density matrices, QEM’s gradient updates are a refinement of updates in both Bai et al. (2009)’s Supervised Semantic Indexing (SSI) and Rocchio (1971); a query concept embedding moves toward the embeddings of relevant documents’ concepts and away from the embeddings of non”
Hash Embeddings for Efficient Word Representations (2017, background): “Conversely, Bai et al. (2009) have experimented with hashing and treating words that co-occur frequently as the same feature in order to reduce dimensionality.”
Up next: retrieval methods for large scale related video suggestion (2014, unknown): “Finally our work is also related to some of the current work on supervised topic weighting for video [31] and text [5] retrieval.”
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (2019, background): “, 1990), supervised embeddings (Bai et al., 2009; Wu et al., 2018) and classical siamese networks (Bromley et al.”
Good Practice in Large-Scale Learning for Image Classification (2014, methodology): “However, we also experimented with a fixed step size as in [2] and [82].”
Towards good practice in large-scale learning for image classification (2012, methodology): “However, we also experimented with a fixed step size η as in [2, 38].”
Real-time Inference in Multi-sentence Tasks with Deep Pretrained Transformers (2019, methodology): “Such methods include vector space models (Salton et al., 1975), LSI (Deerwester et al., 1990), supervised embeddings (Bai et al., 2009) and classical siamese networks (Bromley et al., 1994).”
Learning Concept Embeddings for Query Expansion by Quantum Entropy Minimization (2014, unknown): “2013) and Supervised Semantic Indexing (SSI) (Bai et al. 2009).”
A Deep Architecture for Matching Short Texts (2013, methodology): “Linear models of this direction include the Partial Least Square (PLS) [19] [20], Canonical Correlation Analysis (CCA) [7], and their large margin variants [1].”
Label embedding for text recognition (2013, methodology): “3) show that this similarity is more accurate than the dot product and other explicit methods for learning a similarity function, such as supervised semantic indexing (SSI) [1], in a retrieval task.”