RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words

By Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, Rongrong Ji. 2021

263 citations recorded by the source corpus.

Recent progress on visual question answering has explored the merits of grid features for vision language tasks. Meanwhile, transformer-based models have shown remarkable performance in various sequence prediction problems. However, the spatial information loss of grid features caused by flattening operation, as well as the defect of the transformer model in distinguishing visual words and non visual words, are still left unexplored. In this paper, we first propose Grid-Augmented (GA) module, in which relative geometry features between grids are incorporated to enhance visual representations. Then, we build a BERT-based language model to extract language context and propose Adaptive-Attention (AA) module on top of a transformer decoder to adaptively measure the contribution of visual and language cues before making decisions for word prediction. To prove the generality of our proposals, we apply the two modules to the vanilla transformer model to build our Relationship-Sensitive Transformer (RSTNet) for image captioning task. The proposed model is tested on the MSCOCO benchmark, where it achieves new state-of-art results on both the Karpathy test split and the online test server. Source code is available at GitHub 1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Xiaowei Hu
  6. Giuseppe Fiameni
  7. Jianfeng Gao
  8. Rita Cucchiara

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Zicheng Liu
  3. Lijuan Wang
  4. Rita Cucchiara
  5. Jianfeng Wang
  6. Lorenzo Baraldi
  7. Marcella Cornia
  8. Luc Van Gool

In Multimodal Learning

  1. Lijuan Wang
  2. Jianfeng Wang
  3. Zicheng Liu
  4. Zhe Gan
  5. Xiaowei Hu
  6. Rita Cucchiara
  7. Lorenzo Baraldi
  8. Marcella Cornia

In Natural Language Generation

  1. Rita Cucchiara
  2. Lorenzo Baraldi
  3. Marcella Cornia
  4. Giuseppe Fiameni
  5. Matteo Stefanini
  6. Silvia Cascianelli
  7. Nigel Collier
  8. Radu Soricut

Citations in the corpus