Semantic Regularisation for Recurrent Image Annotation

By Feng Liu, Tao A. Xiang, Timothy M. Hospedales, Wankou Yang, Changyin Sun. 2016

111 citations recorded by the source corpus.

The CNN-RNN design pattern is increasingly widely applied in a variety of image annotation tasks including multi-label classification and captioning. Existing models use the weakly semantic CNN hidden layer or its transform as the image embedding that provides the interface between the CNN and RNN. This leaves the RNN overstretched with two jobs: predicting the visual concepts and modelling their correlations for generating structured annotation output. Importantly this makes the end-to-end training of the CNN and RNN slow and ineffective due to the difficulty of back propagating gradients through the RNN to train the CNN. We propose a simple modification to the design pattern that makes learning more effective and efficient. Specifically, we propose to use a semantically regularised embedding layer as the interface between the CNN and RNN. Regularising the interface can partially or completely decouple the learning problems, allowing each to be more effectively trained and jointly training much more efficient. Extensive experiments show that state-of-the art performance is achieved on multi-label classification as well as image captioning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “To overcome the scalability shortcoming while still modeling label co-occurrences, probabilistic classifier chains [9] and recurrent neural network-based approaches [48, 34, 23, 27] decompose the joint distribution into conditionals at the expense of introducing intrinsic label ordering during train”
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “…2015; Devlin et al., 2015; Donahue et al., 2015a; Karpathy and Li, 2015; Mao et al., 2015; Vinyals et al., 2015; Xu et al., 2015; Yang et al., 2016; Liu et al., 2017a,b), which are mainly differed by the underlying CNN and RNN architectures, and whether or not the attention mechanisms are…”
Attend and Imagine: Multi-Label Image Classification With Visual Attention and Recurrent Neural Networks (2019, methodology): “[38] pre-train a CNN model and a RNN model separately.”
Actor-Critic Sequence Training for Image Captioning (2017, methodology): “Semantic [10]+Attention [26] 0.10 Semantic [10]+Self-critical [17] 0.13 Semantic [10]+Self-critical [17]+Attention [26] 0.18 Ours 0.07 Table 3: Training time for one minibatch on COCO dataset (in seconds)”