Deep Convolutional Ranking for Multilabel Image Annotation

By Yunchao Gong, Yangqing Jia, Thomas K. Leung, Alexander Toshev, Sergey Ioffe. 2013

458 citations recorded by the source corpus.

Multilabel image annotation is one of the most important challenges in computer vision with many real-world applications. While existing work usually use conventional visual features for multilabel annotation, features based on Deep Neural Networks have shown potential to significantly boost performance. In this work, we propose to leverage the advantage of such features and analyze key components that lead to better performances. Specifically, we show that a significant performance gain could be obtained by combining convolutional architectures with approximate top-$k$ ranking objectives, as thye naturally fit the multilabel tagging problem. Our experiments on the NUS-WIDE dataset outperforms the conventional visual features by about 10%, obtaining the best reported performance in the literature.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “j [12, 32] to model the joint distribution of set elements and train a model by minimizing the cross-entropy loss between”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, methodology): “[16] combine deep CNNs with a top-k approximate ranking loss to predict multiple labels.”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “To achieve this, we propose to minimize the following pairwise ranking loss (Gong et al., 2013) with margin δ:”
Learning to See by Moving (2015, background): “Recent advances in computer vision have shown that visual features learnt by training neural networks for the task of object recognition using more than a million labelled images are useful for many computer vision tasks like semantic segmentation, object detection and action classification [22, 13,”
Unsupervised Learning of Visual Representations Using Videos (2015, background): “Finally, our work is also related to metric learning via deep networks [51, 31, 4, 17, 15, 22, 54].”
Image Captioning with Semantic Attention (2016, methodology): “In this paper we mainly investigate two state-of-the-art deep learning models for attribute prediction: using a ranking loss as objective function to learn a multi-label classifier as in [13], and using a Fully Convolutional Network (FCN) [24] to learn attributes from local patches as in [11].”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, methodology): “In addition, the ranking function used in this paper is related to deep metric learning with Siamese architectures [5, 21, 19, 54, 24].”
HCP: A Flexible CNN Framework for Multi-Label Image Classification (2014, background): “[16] studied and compared several multi-label loss functions for the multi-label annotation problem based on a network structure similar to [23].”
Multi-Task CNN Model for Attribute Prediction (2015, methodology): “Because this work is mainly related to the topics of Semantic attributes, Multi-task learning and Deep CNN, we briefly review the most recent literature on these approaches including the following.”
Attentive Recurrent Neural Network for Weak-supervised Multi-label Image Classification (2018, methodology): “Compared with methods using CNN features, such as WARP [8], CNN-Softmax, ResNet-50[10 : Comparison performances on small objects be-tween CNN-RNN [36], RIA [14] and Ours. much better classification performances because our RHN learns the more specific feature representation than methods using the wh”
Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification (2017, methodology): “[11] evaluated various loss functions and found that weighted approximate ranking loss worked best with CNNs.”
Learn to Predict Sets Using Feed-Forward Neural Networks (2020, methodology): “We employ the common evaluation metrics for multi-label image classification also used in [19], [48].”
On the Automatic Generation of Medical Imaging Reports (2017, background): “Note that the loss function of both models are softmax cross-entropy loss [7].”
Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition (2017, methodology): “Speciﬁcally, our model achieves a mAP of (Gong et al. 2013), on this dataset.”
ProjE: Embedding Projection for Knowledge Graph Completion (2016, background): “Recently, softmax regression loss has achieved good results in multi-label image annotation tasks [12, 11].”
Joint Learning of Set Cardinality and State Distribution (2017, background): “One example is Gong et al. (2013b), who combine deep CNNs with a top-k approximate ranking loss to predict multiple labels.”
Deep Classifiers from Image Tags in the Wild (2015, background): “Testing and training LR on NUS produces better scores than training on YFCC100M alone; it also produces better scores than the reported state-of-the-art on this dataset [6].”