Regularizing RNNs for Caption Generation by Reconstructing the Past with the Present
By Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, Wei David Liu. 2018
97 citations recorded by the source corpus.
Recently, caption generation with an encoder-decoder framework has been extensively studied and applied in different domains, such as image captioning, code captioning, and so on. In this paper, we propose a novel architecture, namely Auto-Reconstructor Network (ARNet), which, coupling with the conventional encoder-decoder framework, works in an end-to-end fashion to generate captions. ARNet aims at reconstructing the previous hidden state with the present one, besides behaving as the input-dependent transition operator. Therefore, ARNet encourages the current hidden state to embed more information from the previous one, which can help regularize the transition dynamics of recurrent neural networks (RNNs). Extensive experimental results show that our proposed ARNet boosts the performance over the existing encoder-decoder models on both image captioning and source code captioning tasks. Additionally, ARNet remarkably reduces the discrepancy between training and inference processes for caption generation. Furthermore, the performance on permuted sequential MNIST demonstrates that ARNet can effectively regularize RNN, especially on modeling long-term dependencies. Our code is available at: https://github.com/chenxinpeng/ARNet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Marcella Cornia
- Rita Cucchiara
- Lorenzo Baraldi
- Giuseppe Fiameni
- Matteo Stefanini
- Silvia Cascianelli
- Lin Ma
- Amir Hertz
In Computer Vision and Pattern Recognition
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Silvia Cascianelli
- Wenhao Jiang
- Wei David Liu
In Model Architectures
- Amir Hertz
- Ron Mokady
- Lin Ma
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Giuseppe Fiameni
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Giuseppe Fiameni
- Wenhao Jiang
- Lin Ma
- Ron Mokady
Citations in the corpus
- ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Early works have used LSTM variants [8, 38, 39], while recent works [16, 26] adopted the improved transformer architecture [36].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “visual encoding [40], [43], [46], [47], [48], [49].”
- Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, background): “However, what makes video captioning more challenging than image captioning [12, 44, 7, 17] is not only that the input of video captioning are multiple images, but also that video contains richer semantics, such as spatial/temporal information, content/motion information, and even speech information”
- Recurrent Fusion Network for Image Captioning (2018, unknown): “The encoder-decoder framework, with its advance in machine translation [10, 11], has demonstrated promising performance for image captioning [7, 12, 13].”