Entangled Transformer for Image Captioning
By Guang Li, Linchao Zhu, Ping Liu, Yi Yang. 2019
350 citations recorded by the source corpus.
In image captioning, the typical attention mechanisms are arduous to identify the equivalent visual signals especially when predicting highly abstract words. This phenomenon is known as the semantic gap between vision and language. This problem can be overcome by providing semantic attributes that are homologous to language. Thanks to the inherent recurrent nature and gated operating mechanism, Recurrent Neural Network (RNN) and its variants are the dominating architectures in image captioning. However, when designing elaborate attention mechanisms to integrate visual inputs and semantic attributes, RNN-like variants become unflexible due to their complexities. In this paper, we investigate a Transformer-based sequence modeling framework, built only with attention layers and feedforward layers. To bridge the semantic gap, we introduce EnTangled Attention (ETA) that enables the Transformer to exploit semantic and visual information simultaneously. Furthermore, Gated Bilateral Controller (GBC) is proposed to guide the interactions between the multimodal information. We name our model as ETA-Transformer. Remarkably, ETA-Transformer achieves state-of-the-art performance on the MSCOCO image captioning dataset. The ablation studies validate the improvements of our proposed modules.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Ming-Hsuan Yang
- Cordelia Schmid
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
- Zhe Gan
- Lijuan Wang
- Zicheng Liu
- Cordelia Schmid
- Jianfeng Wang
- Zhengyuan Yang
- Xiaowei Hu
- Rita Cucchiara
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Giuseppe Fiameni
- Arsha Nagrani
- Zhengyuan Yang
- Cordelia Schmid
Citations in the corpus
- Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Besides, researchers have explored to leverage semantic attributes [26,53], scene graphs [50], and graph convolutional networks [52] for captioning.”
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
- Meshed-Memory Transformer for Image Captioning (2019, methodology): “Additive attention or graph-like structures [48] are often added to the recurrence [45, 14] in order to model the relationships between image regions, words, and eventually tags [24].”
- Attention Bottlenecks for Multimodal Fusion (2021, methodology): “Multimodal transformers have been applied to various tasks including audio enhancement [17, 55], speech recognition [25], image segmentation [61, 55], cross-modal sequence generation [40, 38], video retrieval [21] and image/video captioning/classification [43, 54, 53, 37, 28].”
- DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation (2020, methodology): “Inspired by the success of transformer-based architectures for a variety of tasks [17, 10, 1, 22], we also adopt it as the basic building block for our network.”
- Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, background): “have been proposed to achieve the state-of-the-art performance in VQA [2], [6], [7], [48] and image captioning [3], [9], [10], [49]–[53].”
- RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, methodology): “[20] extended the attention module linking transformer encoder and decoder to exploit visual information and semantic knowledge extracted by a external attribute detector.”
- Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, background): “…several works have encoded regions through additive attention [22, 25, 41], graph-based encodings [16, 63, 65], or self-attentive structures [9, 21, 32, 39], the emergence of self-attentive visual encoders [12] and large-scale multi-modal models like CLIP [42] is reopening the discussion on which…”
- A Survey on Vision Transformer (2020, methodology): “[71] exploits the transformer based on the EnTangled Attention (ETA) module to tackle the image captioning task.”
- AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, background): “More recently, with the development of attention mechanism [84], transformer based networks achieve top performance for visual-text [95, 77, 19, 54, 38, 76, 76], visual-audio [26, 89] cross-modal sequence to sequence generation task.”
- Transformers in Medical Image Analysis: A Review (2022, background): “cluding image classification [13], detection [14], segmentation [15], generation [16], and captioning [17].”
- Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning (2020, background): “devise an entangled attention to integrate the visual and semantic modalities into the encoder in [18].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “[76] proposed a gating mechanism for the cross-attention operator, which controls the”
- CPTR: Full Transformer Network for Image Captioning (2021, methodology): “We compare proposed CPTR to “CNN+RNN” models including LSTM [18], SCST [9], LSTM-A [10], RFNet [11], Up- Down [4], GCN-LSTM [12], LBPF [19], SGAE [13] and “CNN+Transformer” models including ORT [20], ETA [14].”