Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification

By Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, Xiaogang Wang. 2017

383 citations recorded by the source corpus.

Multi-label image classification is a fundamental but challenging task in computer vision. Great progress has been achieved by exploiting semantic relations between labels in recent years. However, conventional approaches are unable to model the underlying spatial relations between labels in multi-label images, because spatial annotations of the labels are generally not provided. In this paper, we propose a unified deep neural network that exploits both semantic and spatial relations between labels with only image-level supervisions. Given a multi-label image, our proposed Spatial Regularization Network (SRN) generates attention maps for all labels and captures the underlying relations between them via learnable convolutions. By aggregating the regularized classification results with original results by a ResNet-101 network, the classification performance can be consistently improved. The whole deep neural network is trained end-to-end with only image-level annotations, thus requires no additional efforts on image annotations. Extensive evaluations on 3 public datasets with different types of labels show that our approach significantly outperforms state-of-the-arts and has strong generalization capability. Analysis of the learned SRN model demonstrates that it can effectively capture both semantic and spatial relations of labels for improving classification performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, methodology): “Note that state-of-theart results for NUS-WIDE ignore empty annotations and/or randomly rearrange their splits [6, 61, 24, 27, 29, 23, 57, 12], and thus are not comparable to the results presented in this study.”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “Following the convention of multi-label classification [65,73,24], we report the average overall F1 (O-F1), per-class F1 (C-F1) as well as the mAP.”
Decoupling Category-wise Independence and Relevance with Self-attention for Multi-label Image Classification (2019, methodology): “Note that SRN and MEFF use the same ResNet101 backbone as our models, while ResNet152 is used by OF-RNN as backbone.”
General Multi-label Image Classification with Transformers (2020, background): “Another line of recent works employ object localization techniques[11, 9] or attention mechanism[8, 12] to locate semantic meaningful regions and try to identify underlying relations between regions and outputs.”
The Emerging Trends of Multi-Label Learning (2020, background): “For multi-label image classification, [140] proposes a unified deep neural network that exploits”
Attend and Imagine: Multi-Label Image Classification With Visual Attention and Recurrent Neural Networks (2019, neutral): “We note in [45], Zhu et al.”
Learning Social Image Embedding with Deep Multimodal Attention Networks (2017, methodology): “e aention estimator fa (·) can be calculated in many ways such as CNN in [33].”
Multi-evidence Filtering and Fusion for Multi-label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning (2018, methodology): “Following [47, 42], we employ macro/micro precision, macro/micro recall, and macro/micro F1-measure to evaluate our trained models.”
ELASTIC: Improving CNNs With Dynamic Scaling Policies (2018, methodology): “Following the literature of multilabel classification[41, 7, 39, 17], results are evaluated using macro/micro evaluations.”
ESPNetv2: A Light-Weight, Power Efficient, and General Purpose Convolutional Neural Network (2018, methodology): “Following [65], we evaluated our method on the validation set (40,504 images) using class-wise and overall F1 score.”
Attentive Recurrent Neural Network for Weak-supervised Multi-label Image Classification (2018, methodology): “Our model also outperform SRN ([48], ResNet-based Spatial Regularization Net), which benefit from both the effective feature representation from RHN and the robust label correlation mining from GRRE.”