Recurrent Fusion Network for Image Captioning

By Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei David Liu, T. Zhang. 2018

260 citations recorded by the source corpus.

Recently, much advance has been made in image captioning, and an encoder-decoder framework has been adopted by all the state-of-the-art models. Under this framework, an input image is encoded by a convolutional neural network (CNN) and then translated into natural language with a recurrent neural network (RNN). The existing models counting on this framework employ only one kind of CNNs, e.g., ResNet or Inception-X, which describes the image contents from only one specific view point. Thus, the semantic meaning of the input image cannot be comprehensively understood, which restricts improving the performance. In this paper, to exploit the complementary information from multiple encoders, we propose a novel recurrent fusion network (RFNet) for the image captioning task. The fusion process in our model can exploit the interactions among the outputs of the image encoders and generate new compact and informative representations for the decoder. Experiments on the MSCOCO dataset demonstrate the effectiveness of our proposed RFNet, which sets a new state-of-the-art for image captioning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Rita Cucchiara
  2. Marcella Cornia
  3. Lorenzo Baraldi
  4. Tao Mei
  5. Giuseppe Fiameni
  6. Hanwang Zhang
  7. Ting Yao
  8. Matteo Stefanini

In Computer Vision and Pattern Recognition

  1. Dacheng Tao
  2. Luc Van Gool
  3. Rita Cucchiara
  4. Lorenzo Baraldi
  5. Marcella Cornia
  6. Yi Yang
  7. Xiaogang Wang
  8. Trevor Darrell

In Multimodal Learning

  1. Rita Cucchiara
  2. Lorenzo Baraldi
  3. Marcella Cornia
  4. Giuseppe Fiameni
  5. Matteo Stefanini
  6. Tao Mei
  7. Hanwang Zhang
  8. Ting Yao

In Natural Language Generation

  1. Rita Cucchiara
  2. Lorenzo Baraldi
  3. Marcella Cornia
  4. Matteo Stefanini
  5. Giuseppe Fiameni
  6. Wenhao Jiang
  7. Tao Mei
  8. Lin Ma

Citations in the corpus