TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation

By Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek, Cordelia Schmid. 2009

764 citations recorded by the source corpus.

Image auto-annotation is an important open problem in computer vision. For this task we propose TagProp, a discriminatively trained nearest neighbor model. Tags of test images are predicted using a weighted nearest-neighbor model to exploit labeled training images. Neighbor weights are based on neighbor rank or distance. TagProp allows the integration of metric learning by directly maximizing the log-likelihood of the tag predictions in the training set. In this manner, we can optimally combine a collection of image similarity metrics that cover different aspects of image content, such as local shape descriptors, or global color histograms. We also introduce a word specific sigmoidal modulation of the weighted neighbor tag predictions to boost the recall of rare words. We investigate the performance of different variants of our model and compare to existing work. We present experimental results for three challenging data sets. On all three, TagProp makes a marked improvement as compared to the current state-of-the-art.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, background): “There have been many works handling the vital problem of multi-label classification [59,32].”
Deep Sets (2017, methodology): “For the above two datasets, train/test splits are similar to those used in previous works (Guillaumin et al., 2009; Chen et al., 2013).”
Choosing Linguistics over Vision to Describe Images (2012, neutral): “Phrase Annotation Another interesting application of our approach is phrase annotation ; i.e., we can annotate images with phrase(s) instead of just keywords (Makadia, Pavlovic, and Kumar 2010; Guillaumin et al. 2009).”
Deep Convolutional Ranking for Multilabel Image Annotation (2013, methodology): “Multilabel image annotation [25, 14] is an important and challenging problem in computer vision.”
A Survey on Metric Learning for Feature Vectors and Structured Data (2013, background): “…vision problems, such as image classification (Mensink et al., 2012), object recognition (Frome et al., 2007; Verma et al., 2012), face recognition (Guillaumin et al., 2009b; Lu et al., 2012), visual tracking (Li et al., 2012; Jiang et al., 2012) or image annotation (Guillaumin et al., 2009a).”
Attentive Recurrent Neural Network for Weak-supervised Multi-label Image Classification (2018, methodology): “For fair comparison, we follow previous work [9] to evaluate the performance.”
Learning Attributes Equals Multi-Source Domain Generalization (2016, methodology): “We input our learned feature representations to two popular frameworks for multi-attribute based image retrieval: TagProp [28] and the fusion of individual prediction scores [42].”
ImageNet Auto-Annotation with Segmentation Propagation (2014, background): “This is related to works that transfer annotations between images based on their global similarity, [51,43,30,31,62,54] as done in inpainting [31], image tagging [30], object class detection [54], and scene parsing [43,62].”
Multilabel Image Classification With Regional Latent Semantic Dependencies (2016, background): “The most direct approach is to treat multi-label image classification problem as several separate single label classification problems, and to train the independent classifier for each label with a cross-entropy [3] or ranking loss (such as WARP [8]).”
Graph-RISE: Graph-Regularized Image Semantic Embedding (2019, background): “There are several prior works on learning image similarity [8, 26, 28].”
Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost (2013, neutral): “New images (of new classes) are simply added to the database, and can be used for classiﬁcation without further processing.”
Content-based visual search learned from social media (2012, methodology): “A considerable amount of papers have been published for generic image annotation [41,57,76,79,103,104,115,134].”