Entangled Transformer for Image Captioning

By Guang Li, Linchao Zhu, Ping Liu, Yi Yang. 2019

350 citations recorded by the source corpus.

In image captioning, the typical attention mechanisms are arduous to identify the equivalent visual signals especially when predicting highly abstract words. This phenomenon is known as the semantic gap between vision and language. This problem can be overcome by providing semantic attributes that are homologous to language. Thanks to the inherent recurrent nature and gated operating mechanism, Recurrent Neural Network (RNN) and its variants are the dominating architectures in image captioning. However, when designing elaborate attention mechanisms to integrate visual inputs and semantic attributes, RNN-like variants become unflexible due to their complexities. In this paper, we investigate a Transformer-based sequence modeling framework, built only with attention layers and feedforward layers. To bridge the semantic gap, we introduce EnTangled Attention (ETA) that enables the Transformer to exploit semantic and visual information simultaneously. Furthermore, Gated Bilateral Controller (GBC) is proposed to guide the interactions between the multimodal information. We name our model as ETA-Transformer. Remarkably, ETA-Transformer achieves state-of-the-art performance on the MSCOCO image captioning dataset. The ablation studies validate the improvements of our proposed modules.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Dacheng Tao
  2. Luc Van Gool
  3. Ming-Hsuan Yang
  4. Cordelia Schmid
  5. Xiaogang Wang
  6. Trevor Darrell
  7. Jitendra Malik
  8. Andrew Zisserman

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Cordelia Schmid
  5. Jianfeng Wang
  6. Zhengyuan Yang
  7. Xiaowei Hu
  8. Rita Cucchiara

In Natural Language Generation

  1. Rita Cucchiara
  2. Lorenzo Baraldi
  3. Marcella Cornia
  4. Matteo Stefanini
  5. Giuseppe Fiameni
  6. Arsha Nagrani
  7. Zhengyuan Yang
  8. Cordelia Schmid

Citations in the corpus