Attend and Imagine: Multi-Label Image Classification With Visual Attention and Recurrent Neural Networks

By Fan Lyu, Qi Wu, Fuyuan Hu, Qingyao Wu, Mingkui Tan. 2019

55 citations recorded by the source corpus.

Real images often have multiple labels, i.e., each image is associated with multiple objects or attributes. Compared to single-label image classification, the multilabel classification problem is much more challenging due to several issues. At first, multiple objects can be anywhere in the image. Second, the importance of different regions in an image is different, and the regions of interest in a multilabel image can be very different from another one. Finally, multiple labels of an image can have label dependencies due to complex image structures. To address these challenges, in this paper, we propose to predict the labels sequentially by applying the recurrent neural networks (RNNs), which are used to encode the label dependencies. When predicting a specific label, we introduce a dynamic attention mechanism to enable the model to focus on only regions of interest in the image. Two benchmark datasets (i.e., Pascal VOC and MS-COCO) are adopted to demonstrate the effectiveness of our work. Moreover, we construct a new dataset, which includes many semantic dependent labels in each image, to verify the effectiveness of our model. Experimental results show that our method outperforms several state-of-the-arts, especially when predicting some semantic relative labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zihan Ye
2. Fan Lyu
3. Bin Sheng
4. D. Feng
5. Song Wang
6. Shuai Wang
7. Fuyuan Hu
8. Wei Feng

In Computer Vision and Pattern Recognition

1. Zihan Ye
2. Fan Lyu
3. Bin Sheng
4. Ping Li
5. D. Feng
6. Song Wang
7. Shuai Wang
8. Shuzhou Sun

In Model Architectures

1. Fan Lyu
2. Bin Sheng
3. D. Feng
4. Fuyuan Hu
5. Wei Feng

Citations in the corpus

• Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, neutral): “In order to explicitly exploit label co-occurrences, researchers have resorted to modeling the joint probability distribution of labels [59] or decomposing the joint distribution into conditionals [11, 61, 45, 28, 32, 41, 46].”
• EAPT: Efficient Attention Pyramid Transformer for Image Processing (2021, methodology): “The main difference between Transformer and RNNs is that the former is parallel while the latter is not.”
• Multi-Domain Multi-Task Rehearsal for Lifelong Learning (2020, background): “In contrast to static machine learning (He et al. 2016; Deng et al. 2018; Lyu et al. 2019; Lyu, Feng, and Wang 2020), Lifelong Learning (Ring 1997; Thrun 1998) is proposed to improve the self-learning ability of the machine that continually learns new knowledge.”
• AGCN: Augmented Graph Convolutional Network for Lifelong Multi-Label Image Recognition (2022, unknown): “Some methods [17, 18, 19] used recurrent neural network multi-label recognition under a restrictive assumption that the label relationships are in order, which limited the complex relationships in label space.”