CNN-RNN: A Unified Framework for Multi-label Image Classification

By Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, Wei Xu. 2016

1,254 citations recorded by the source corpus.

While deep convolutional neural networks (CNNs) have shown a great success in single-label image classification, it is important to note that real world images generally contain multiple labels, which could correspond to different objects, scenes, actions and attributes in an image. Traditional approaches to multi-label image classification learn independent classifiers for each category and employ ranking or thresholding on the classification results. These techniques, although working well, fail to explicitly exploit the label dependencies in an image. In this paper, we utilize recurrent neural networks (RNNs) to address this problem. Combined with CNNs, the proposed CNN-RNN framework learns a joint image-label embedding to characterize the semantic label dependency as well as the image-label relevance, and it can be trained end-to-end from scratch to integrate both information in a unified framework. Experimental results on public benchmark datasets demonstrate that the proposed architecture achieves better performance than the state-of-the-art multi-label classification models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “2016 [61] VGG 7 82783 n/a 40504 63.”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, methodology): “In this experiment, similar to [47], we build on the 16-layers VGG network [39], pretrained on the 2012 ImageNet dataset.”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “Recently, recurrent approaches [84,87] and attention-based methods [97,33] are proposed to correlate the labels during predictions.”
Neural Motifs: Scene Graph Parsing with Global Context (2017, neutral): “cation [46].”
Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, methodology): “…of the proposed method against the following state-of-the-art approaches: HCP-2000 [Wei et al., 2014], HCP-VGG [Wei et al., 2016], RLSD+ft-RPN [Zhang et al., 2018], CNN+RNN [Wang et al., 2016], CNN+LSTM [Wang et al., 2017], PF-DLDL [Gao et al., 2017a] and the method of [Zhou et al., 2018].”
Learn to Predict Sets Using Feed-Forward Neural Networks (2020, methodology): “C-P C-R C-F1 O-P O-R O-F1 I-P I-R I-F1 Tensor-B/L (Softmax) Full VGG16 k=k (1) 88.2 65.4 75.1 91.3 59.2 71.8 91.3 69.8 79.1 Tensor-B/L (BCE) Full VGG16 k=k (1) 88.7 58.6 70.5 92.2 59.8 72.5 92.2 70.1 79.6 Set-DS (BCE-NB) [9] Full VGG16 k=m 76.8 74.8 75.8 80.6 76.7 78.6 83.4 81.9 82.6 Set-DS (BCE-Sft”
Scaling Out-of-Distribution Detection for Real-World Settings (2022, background): “Prior work addresses multi-label classification in various ways, such as by leveraging label dependencies [35], but none have explored using multi-label classifiers for out-of-distribution detection.”
Multi-evidence Filtering and Fusion for Multi-label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning (2018, neutral): “CNN-RNN[41] - - - - - - 60.”
Semantic Regularisation for Recurrent Image Annotation (2016, methodology): “Ie could either be a feature layer [14, 22, 31], e.”
Seq2Slate: Re-ranking and Slate Optimization with RNNs (2018, methodology): “…models in complex prediction tasks, including machine translation [Sutskever et al., 2014, Bahdanau et al., 2015], parsing [Vinyals et al., 2015a], combinatorial optimization [Vinyals et al., 2015b, Bello et al., 2017], multi-label classiﬁcation [Wang et al., 2016, Nam et al., 2017], and others.”
CU-Net: Component Unmixing Network for Textile Fiber Identification (2019, methodology): “We compared our methods (CU-Net, CU-Net SI RI , CU-Net RI and CU-Net SI ) with SB-MLC-Net and the state-of-the-art methods (Wang et al. 2016; Yeh et al. 2017).”
Learning Deep Latent Spaces for Multi-Label Classification (2017, methodology): “For example, CNN-RNN (Wang et al. 2016) chose to learn a linear label embedding function, with label coProceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17)”
Maximizing Subset Accuracy with Recurrent Neural Networks in Multi-label Classification (2017, background): “[31] have shown that the descending label ordering allows to achieve a decent performance on multi-label image datasets.”
Fake News Identification on Twitter with Hybrid CNN and RNN Models (2018, methodology): “Although this hybrid approach in semantic interpretation of text and images is not new [14][15], to the best of our knowledge, this is the first attempt involving the use of a hybrid approach in the detection of the origin and propagation of fake news posts.”
Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection (2018, methodology): “CNN-RNN [32] utilizes RNN combined with CNN to learn a joint image-label embedding for capturing label dependencies.”
Multi-view Adversarially Learned Inference for Cross-domain Joint Distribution Matching (2018, background): “, (1) in image translation/editing [8, 16, 21, 30, 32], the goal is to transform an input image into an output one with desired properties; (2) in image annotating/attribute predicting [4, 5, 28, 29], one aims to extract meaningful semantic labels as many as possible from a given image; (3) in attri”
Understanding and Controlling User Linkability in Decentralized Learning (2018, methodology): “, [19, 40, 54, 73] for AP and [16, 29, 38] for Top-1/5 accuracy.”