Multi-label Image Recognition by Recurrently Discovering Attentional Regions

By Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, Liang Lin. 2017

334 citations recorded by the source corpus.

This paper proposes a novel deep architecture to address multi-label image recognition, a fundamental and practical task towards general visual understanding. Current solutions for this task usually rely on an extra step of extracting hypothesis regions (i.e., region proposals), resulting in redundant computation and sub-optimal performance. In this work, we achieve the interpretable and contextualized multi-label image classification by developing a recurrent memorized-attention module. This module consists of two alternately performed components: i) a spatial transformer layer to locate attentional regions from the convolutional feature maps in a region-proposal-free way and ii) an LSTM (Long-Short Term Memory) sub-network to sequentially predict semantic labeling scores on the located regions while capturing the global dependencies of these regions. The LSTM also output the parameters for computing the spatial transformer. On large-scale benchmarks of multi-label image classification (e.g., MS-COCO and PASCAL VOC 07), our approach demonstrates superior performances over other existing state-of-the-arts in both accuracy and efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “Instead, they care about evaluating their topk predictions [12, 57, 7, 49, 48], by manually fixing k for all samples, or apply a fixed threshold to label probabilities [61, 5] (allowing for different number of images per sample).”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “Recently, recurrent approaches [84,87] and attention-based methods [97,33] are proposed to correlate the labels during predictions.”
Deep Reasoning with Knowledge Graph for Social Relationship Understanding (2018, background): “[Wang et al. , 2017] exploits LSTM network to capture label dependencies by remembering previous information step by step.”
Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networks (2017, background): “For example, the VGG16-Net (Simonyan and Zisserman 2014), which has demonstrated remarkable performance on a wide range of recognition tasks (Long, Shelhamer, and Darrell 2015; Ren et al. 2015; Wang et al. 2017), requires about 15.”
General Multi-label Image Classification with Transformers (2020, background): “Multi-label image classi-ﬁcation is a visual recognition task that aims to predict a set of labels corresponding to objects, attributes, or actions given an input image [15, 46, 48, 50, 6, 32, 9].”
Multi-evidence Filtering and Fusion for Multi-label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning (2018, methodology): “Following [47, 42], we employ macro/micro precision, macro/micro recall, and macro/micro F1-measure to evaluate our trained models.”
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding (2018, methodology): “Recently, visual attentionmodels [5, 26, 30, 42] were intensively proposed to automatically search the informative regions and various works successfully applied this technique to FGIR [8, 17, 28, 50].”
Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up (2019, methodology): “Attentional regions is discovered recurrently by incorporating a LSTM sub-network for multi-label image classification in [46].”
Decoupling Category-wise Independence and Relevance with Self-attention for Multi-label Image Classification (2019, methodology): “In addition to two baselines by using ResNet101 and WILDCAT, we further compare DecoupleNet with recent state-of-the-art approaches, including WARP [9], CNNRNN [4], RLSD [22], RNN-Attention [7], RNN-RL [23], OF-RNN [6], SRN [8] and MEFF [24].”
Asymmetric Loss For Multi-Label Classification (2020, methodology): “Other approaches are based on modeling image parts and attentional regions [37, 13, 33, 36], as well as using recurrent neural networks [25, 30].”
Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition (2019, background): “[26] introduced spatial transformer to adaptively search semantic-aware regions and then aggregated features from these regions to identify multiple labels.”
Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers (2019, neutral): “, 2016], CNN+LSTM [Wang et al., 2017], PF-DLDL [Gao et al.”
Multi-Label Image Recognition With Graph Convolutional Networks (2019, methodology): “Another line of works implicitly model the label correlations via attention mechanisms [36, 29].”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, methodology): “CNN-RNN [33] utilized the RNNs combined with CNN to learn a joint image-label embedding, and Wang et al [35] took advantages of a spatial transformer layer and long short-term memory(LSTM) units to capture contextual dependencies.”