SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning

By Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei David Liu, Tat-seng Chua. 2016

1,836 citations recorded by the source corpus.

Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism — a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Exploring Self-Attention for Image Recognition (2020, methodology): “Other approaches [4, 36, 8] adopted both spatial and channel attention.”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “To further leverage the visual cues, an attention mechanism is usually utilized [4,6,42] to focus on specific visual features.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
Attention on Attention for Image Captioning (2019, background): “There are other attention mechanisms such as: spatial and channel-wise attention [6], adaptive attention [27], stacked attention [46], multi-level attention [51], multi-head attention and self-attention [35].”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “A common approach for these tasks is to use temporal models over convolutional features [36, 70, 7].”
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks (2018, background): “Recent work has developed both powerful and generic parameterised attention modules to allow the system to extract informative signals dynamically [46, 4, 45].”
BAM: Bottleneck Attention Module (2018, background): “In fact, this design follows the similar aspect of a human visual system, which has ‘what’ (channel) and ‘where’ (spatial) pathways and both pathways contribute to process visual information [6, 31].”
CBAM: Convolutional Block Attention Module (2018, background): “They also miss the spatial attention, which plays an important role in deciding ‘where’ to focus as shown in [29].”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, methodology): “Instead, our work is inspired by the concise spirit of SNAIL [29], thus, incorporating class-specific soft-attention vectors to achieve channelwise feature selection on each RoI feature in {zi,j} j=1 [5].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “[21, 59, 61, 144, 150, 152] have performed experiments using the dataset.”
Visual Entailment Task for Visually-Grounded Language Learning (2018, methodology): “Image Captioning: Before VE, there are many captioning models [Karpathy and Fei-Fei, 2015, Vinyals et al., 2017, Chen et al., 2017] which can serve as a useful baseline by generating an image caption as the premise and then apply existing TE models for classification.”
Deep Multimodal Representation Learning: A Survey (2019, background): “[162] proposed to incorporate spatial and channel wise attentions in a CNNnetwork.”
STG2Seq: Spatial-temporal Graph to Sequence Model for Multi-step Passenger Demand Forecasting (2019, methodology): “The joint representation Y h + q is then transformed by the importance score α : Inspired by [ Chen et al. , 2017 ] which showed that the importance of each channel is also different, we further add a channel attention module after the temporal attention module to ﬁnd the most important frames in…”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “Model fvqa typically contains an attention mechanism [13, 29, 41], and it is trained with cross-entropy loss [38, 11].”
Self-attention with Functional Time Representation Learning (2019, methodology): “It works by capturing the importance weights of the sequential inputs successively and is often used as an add-on component to base models such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs) [3].”
Dual Contrastive Loss and Attention for GANs (2021, background): “Attention models have dominated the language modeling [78, 86, 17, 19, 89], and became popular among various computer vision problems from image recognition [16, 79, 31, 32, 104, 109, 30, 85] to image cap-tioning [88, 90, 7] to video prediction [37, 83].”