Recurrent Fusion Network for Image Captioning
By Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei David Liu, T. Zhang. 2018
260 citations recorded by the source corpus.
Recently, much advance has been made in image captioning, and an encoder-decoder framework has been adopted by all the state-of-the-art models. Under this framework, an input image is encoded by a convolutional neural network (CNN) and then translated into natural language with a recurrent neural network (RNN). The existing models counting on this framework employ only one kind of CNNs, e.g., ResNet or Inception-X, which describes the image contents from only one specific view point. Thus, the semantic meaning of the input image cannot be comprehensively understood, which restricts improving the performance. In this paper, to exploit the complementary information from multiple encoders, we propose a novel recurrent fusion network (RFNet) for the image captioning task. The fusion process in our model can exploit the interactions among the outputs of the image encoders and generate new compact and informative representations for the decoder. Experiments on the MSCOCO dataset demonstrate the effectiveness of our proposed RFNet, which sets a new state-of-the-art for image captioning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Rita Cucchiara
- Marcella Cornia
- Lorenzo Baraldi
- Tao Mei
- Giuseppe Fiameni
- Hanwang Zhang
- Ting Yao
- Matteo Stefanini
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Yi Yang
- Xiaogang Wang
- Trevor Darrell
In Multimodal Learning
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Tao Mei
- Hanwang Zhang
- Ting Yao
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Giuseppe Fiameni
- Wenhao Jiang
- Tao Mei
- Lin Ma
Citations in the corpus
- Attention on Attention for Image Captioning (2019, methodology): “The models include: LSTM [37], which encodes the image using CNN and decodes it using LSTM; SCST [31], which employs a modified visual attention and is the first to use SCST to directly optimize the evaluation metrics; Up-Down [2], which employs a two-LSTM layer model with bottom-up features extract”
- SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “[13] proposed a recurrent fusion network to integrate the output of multiple image encoders.”
- Meshed-Memory Transformer for Image Captioning (2019, methodology): “Also, we compare to RFNet [15], which uses a recurrent fusion network to merge different CNN features; GCN-LSTM [48], which exploits pairwise relationships between image regions through a Graph CNN; SGAE [46], which instead uses auto-encoding scene graphs.”
- Aligning Visual Regions and Textual Concepts: Learning Fine-Grained Image Representations for Image Captioning (2019, background): “, 2017), and has recently shown great success in image captioning (Anderson et al., 2018; Jiang et al., 2018; Yao et al., 2018).”
- Multimodal Transformer With Multi-View Visual Representation for Image Captioning (2019, background): “Note that the RFNet [49] also incorporates multi-view features, and they introduce more views than our approach (4 vs. 2).”
- Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, background): “Image captioning [3, 20, 17, 7, 51] aims to generate a natural language sentence to describe the image content.”
- Exploring Hate Speech Detection in Multimodal Publications (2019, methodology): “This approach is applied in tasks such as image captioning [14, 13] and multimodal image retrieval [9, 8].”
- X-Linear Attention Networks for Image Captioning (2020, unknown): “5 RFNet (ResNet+DenseNet+Inception) [13] 80.”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, unknown): “[52] proposed to use multiple CNNs in order to exploit their complementary information, then fused their representations with a recurrent procedure.”
- Hierarchy Parsing for Image Captioning (2019, background): “Furthermore, GCNLSTM by exploring the relations between objects to enrich region-level features, improves SCST, ADP-ATT, UpDown, and RFNet.”
- Deconfounded Image Captioning: A Causal Retrospect (2020, background): “4 to group the compared state of the art captioners into four groups: large-scale pre-training: Up-Down [24], UD-HIP [33], and VLP [35]; attention mechanisms: CAVP [85], RFNet [83], LBPF [84], and AoANet [25]; sentence patterns: CNM [43]; and structured attention: GCN-LSTM [44] and SGAE [45].”
- A Better Variant of Self-Critical Sequence Training (2020, background): “…captioning task, SCST not only has become the new standard for training captioning models (Yao et al., 2017; Dognin et al., 2019; Luo et al., 2018; Jiang et al., 2018; Liu et al., 2018b; Ling and Fi-dler, 2017; Liu et al., 2018a; Chen et al., 2019; Yang et al., 2019; Zhao et al., 2018), but also…”
- Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, background): “However, what makes video captioning more challenging than image captioning [12, 44, 7, 17] is not only that the input of video captioning are multiple images, but also that video contains richer semantics, such as spatial/temporal information, content/motion information, and even speech information”
- RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, background): “[16], GCN-LSTM [43], SGAE [42], ORT [12], AoANet [13], M(2) Transformer [6], and X-Transformer [28].”
- Learning to Collocate Neural Modules for Image Captioning (2019, result): “All of StackCap, CAVP, and RFNet use wider encoders or deeper decoders, while they do not design different modules.”
- Unpaired Image-to-Speech Synthesis With Multimodal Information Bottleneck (2019, methodology): “For fair comparisons, we replace the 3-layer CNN with RFNet (Ours w/ [20]) and UD (Ours w/ [3]).”
- From Show to Tell: A Survey on Image Captioning (2021, methodology): “On the same line, Jiang et al. [40] proposed to use multiple CNNs in order to exploit their complementary information, then fused their representations with a recurrent procedure.”