X-Linear Attention Networks for Image Captioning

By Yingwei Pan, Ting Yao, Yehao Li, Tao Mei. 2020

635 citations recorded by the source corpus.

Recent progress on fine-grained visual recognition and visual question answering has featured Bilinear Pooling, which effectively models the 2nd order interactions across multi-modal inputs. Nevertheless, there has not been evidence in support of building such interactions concurrently with attention mechanism for image captioning. In this paper, we introduce a unified attention block --- X-Linear attention block, that fully employs bilinear pooling to selectively capitalize on visual information or perform multi-modal reasoning. Technically, X-Linear attention block simultaneously exploits both the spatial and channel-wise bilinear attention distributions to capture the 2$^{nd}$ order interactions between the input single-modal or multi-modal features. Higher and even infinity order feature interactions are readily modeled through stacking multiple X-Linear attention blocks and equipping the block with Exponential Linear Unit (ELU) in a parameter-free fashion, respectively. Furthermore, we present X-Linear Attention Networks (dubbed as X-LAN) that novelly integrates X-Linear attention block(s) into image encoder and sentence decoder of image captioning model to leverage higher order intra- and inter-modal interactions. The experiments on COCO benchmark demonstrate that our X-LAN obtains to-date the best published CIDEr performance of 132.0% on COCO Karpathy test split. When further endowing Transformer with X-Linear attention blocks, CIDEr is boosted up to 132.8%. Source code is available at https://github.com/Panda-Peter/image-captioning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Scaling Up Vision-Language Pretraining for Image Captioning (2021, neutral): “Recent works have proposed enhanced attention-based models to improve the performance, such as ORT [14], AoANet [16], M 2 Transformer [8], X-LAN [37], and RSTNet [57].”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “• X-Transformer [53], which employs bilinear pooling to selectively capitalize on visual information and is one of best-performing models on MS COCO.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “For example, Song et al. (2019); Wang, Chen, and Hu (2019); Gao et al. (2019); Huang et al. (2019); Pan et al. (2020); Guo et al. (2020); Cornia et al. (2020) explore different attention mechanisms in captioning modeling.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “[80] proposed to use bilinear pooling techniques to strengthen the representative capacity of the output attended feature.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “In the decoders, convolutional networks are (a) Efficient Attention [156] (b) Slot Attention [119] (c) RFA [140] (d) X-Linear [135] (e) Axial [171] (f) Transformer [165] Figure 5: The architectures of self-attention methods: Transformers [165], Axial attention [171], X-Linear [135], Slot [119] and R”
TransVG: End-to-End Visual Grounding with Transformers (2021, background): “Multi-modal fusion and reasoning is widely studied in the literature [1, 35, 49, 54, 65], and it is the core problem in visual grounding.”
Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, methodology): “While several works have encoded regions through additive attention [22, 25, 41], graph-based encodings [16, 63, 65], or self-attentive structures [9, 21, 32, 39], the emergence of self-attentive visual encoders [12] and large-scale multi-modal models like CLIP [42] is reopening the discussion on wh”
Text Augmentation Using BERT for Image Captioning (2020, result): “Additionally, we have confirmed our results on the other state-of-the-art models, such as [41,42], on the best variant of dataset chosen based on the previous experiments with [40].”
Dual-Level Collaborative Transformer for Image Captioning (2021, methodology): “To explain, most recent methods in image captioning (Huang et al. 2019; Cornia et al. 2020; Pan et al. 2020) use the self-attention modules to model the relationships of visual features.”
A Unified Sequence Interface for Vision Tasks (2022, methodology): “The above methods cannot do image captioning, so we train a Transformer-based caption model [40, 32] which is specialized for the task.”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, neutral): “Subsequently, Transformers modeled interactions among all image elements with self-attention [17,51,60,65].”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, background): “have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, background): “Referring expression comprehension (REC) [8, 35, 50, 51, 54] is a challenging task that aims at localizing the target object specified by a given referring expression, which can underpin a diverse set of vision-language tasks such as visual question answering [1, 13, 61], image captioning [39, 53], ”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, background): “Despite these signiﬁcant advances, most of the main-stream captioning models [11,53,63,81] rely heavily on a bulky object detector to provide regional visual representations for the multimodal interaction, as shown in Figure 1-a.”