RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words
By Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, Rongrong Ji. 2021
263 citations recorded by the source corpus.
Recent progress on visual question answering has explored the merits of grid features for vision language tasks. Meanwhile, transformer-based models have shown remarkable performance in various sequence prediction problems. However, the spatial information loss of grid features caused by flattening operation, as well as the defect of the transformer model in distinguishing visual words and non visual words, are still left unexplored. In this paper, we first propose Grid-Augmented (GA) module, in which relative geometry features between grids are incorporated to enhance visual representations. Then, we build a BERT-based language model to extract language context and propose Adaptive-Attention (AA) module on top of a transformer decoder to adaptively measure the contribution of visual and language cues before making decisions for word prediction. To prove the generality of our proposals, we apply the two modules to the vanilla transformer model to build our Relationship-Sensitive Transformer (RSTNet) for image captioning task. The proposed model is tested on the MSCOCO benchmark, where it achieves new state-of-art results on both the Karpathy test split and the online test server. Source code is available at GitHub 1.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zhe Gan
- Lijuan Wang
- Zicheng Liu
- Jianfeng Wang
- Xiaowei Hu
- Giuseppe Fiameni
- Jianfeng Gao
- Rita Cucchiara
In Computer Vision and Pattern Recognition
- Zhe Gan
- Zicheng Liu
- Lijuan Wang
- Rita Cucchiara
- Jianfeng Wang
- Lorenzo Baraldi
- Marcella Cornia
- Luc Van Gool
In Multimodal Learning
- Lijuan Wang
- Jianfeng Wang
- Zicheng Liu
- Zhe Gan
- Xiaowei Hu
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Silvia Cascianelli
- Nigel Collier
- Radu Soricut
Citations in the corpus
- Scaling Up Vision-Language Pretraining for Image Captioning (2021, neutral): “Recent works have proposed enhanced attention-based models to improve the performance, such as ORT [14], AoANet [16], M 2 Transformer [8], X-LAN [37], and RSTNet [57].”
- Injecting Semantic Concepts into End-to-End Image Captioning (2021, background): “Some recent works [4,81] arrive at an empirical conclusion that a strong object detector is necessary, providing clean and unambiguous regional features for objects.”
- From Show to Tell: A Survey on Image Captioning (2021, methodology): “Finally, the approach proposed by Zhang et al. [74] completely disregards region features and applies self-attention directly to grid features, incorporating their relative geometry relationships into self-attention computation.”
- Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “After that, the main focus of following methods is to model the interaction between visual and textual cues via attention mechanism to get more faithful and richer captions [85, 46, 3, 91, 27].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “[89] applies selfattention directly to grid features, incorporating their relative geometry relationships into self-attention computation.”