A Discriminative Kernel-Based Approach to Rank Images from Text Queries

By David Grangier, Samy Bengio. 2008

334 citations recorded by the source corpus.

This paper introduces a discriminative model for the retrieval of images from text queries. Our approach formalizes the retrieval task as a ranking problem, and introduces a learning procedure optimizing a criterion related to the ranking performance. The proposed model hence addresses the retrieval problem directly and does not rely on an intermediate image annotation task, which contrasts with previous research. Moreover, our learning procedure builds upon recent work on the online learning of kernel-based classifiers. This yields an efficient, scalable algorithm, which can benefit from recent kernels developed for image comparison. The experiments performed over stock photography data show the advantage of our discriminative ranking approach over state-of-the-art alternatives (e.g. our model yields 26.3% average precision over the Corel dataset, which should be compared to 22.0%, for the best alternative model evaluated). Further analysis of the results shows that our model is especially advantageous over difficult queries such as queries with few relevant pictures or multiple-word queries.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Information Retrieval & Search

Citations in the corpus

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “Since bag-of-words representations have been successfully used for other tasks involving text and images (e.g., Grangier & Bengio, 2008; Hardoon et al., 2006), we include a basic bag of words kernel, which ignores word order and represents each caption simply as a vector of word frequencies.”
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics (2012, methodology): “The data-driven image annotation approaches of Guillaumin et al. (2009), Makadia et al.”
YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video (2017, background): “Some of the larger existing vision data sets rely indirectly on aggregate measurements of human preference [18].”
A survey of approaches and trends in person re-identification (2014, neutral): “Some other variations of Re-ID are verification and searching of people based on textual queries [115,116].”
A Survey of Multi-View Representation Learning (2016, positive): “Further, to exploit the advantage of online learning of kernelbased classifiers, Grangier and Bengio [140] propose a discriminative cross-modal ranking model called Passive-Aggressive Model for Image Retrieval (PAMIR), which not only adopts a learning criterion related to the final retrieval perform”
Large Scale Online Learning of Image Similarity Through Ranking (2009, methodology): “This approach has been found successful for the task of content based image ranking described by Grangier and Bengio (2008).”
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (2018, methodology): “Some commonly used datasets couple images with a handful of un-ordered tags (Barnard et al., 2003; Cusano et al., 2004; Grangier and Bengio, 2008; Chen et al., 2013, inter alia) or short, literal natural language captions (Farhadi et al., 2010; Ord´o˜nez et al., 2011; Kulkarni et al., 2013; Fang et…”
Large scale image annotation: learning to rank with joint word-image embeddings (2010, methodology): “Embedding words in a low dimensional space to capture semantics is a classic (unsupervised) approach in text retrieval which has been adapted for image annotation before, for example PLSA has been used for images [14] but has been shown to perform worse than (non-embedding based) supervised ranking ”
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges (2017, methodology): “Existing methods mainly involve only two media types such as [56], [57], and [59], and when the number of media types increases, there remains a problem on how to incorporate the ranking information of more than two media types into a unified framework.”
Supervised semantic indexing (2009, methodology): “The authors of [17] used a related model to the ones we describe, but for the task of image retrieval, and [15] also used a related (regression-based) method for advert placement.”
Content-based visual search learned from social media (2012, background): “However, since the tag ranking criterion is not directly related to the performance of image retrieval using the tagging results, optimizing image tagging does not necessarily yield good image rankings [40].”
Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment (2015, methodology): “Furthermore, for cross-modal learning to rank, most of the proposed methods such as Passive-Aggressive Model for Image Retrieval (PAMIR) [8] and Supervised Semantic Indexing (SSI) [1] use global alignment to learn the common space so that text-image relevance is measured based on their global…”
Training highly multiclass classifiers (2014, methodology): “Next, we review two stochastic gradient approaches that correspond to different loss functions: AUC sampling (Grangier and Bengio, 2008) and the WARP sampling used in the Wsabie classifier (Weston et al., 2011).”
Image ranking and retrieval based on multi-attribute queries (2011, background): “Notable among them are PAMIR [8] and TagProp [9].”