Long-term recurrent convolutional networks for visual recognition and description
By Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Trevor Darrell. 2014
6,439 citations recorded by the source corpus.
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or “temporally deep”, are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are “doubly deep” in that they can be compositional in spatial and temporal “layers”. Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Trevor Darrell
- Quoc V. Le
- Percy Liang
- Dacheng Tao
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Alan L. Yuille
- Ruslan Salakhutdinov
- Kaiming He
- Vladlen Koltun
- Jianfeng Gao
- Trevor Darrell
Citations in the corpus
- What makes ImageNet good for transfer learning? (2016, methodology): “Using ImageNet pre-trained CNN features, impressive results have been obtained on several image classification datasets [13, 35], object detection [16, 39], action recognition [41], human pose estimation [9], image segmentation [10], optical flow [47], image captioning [12, 22] and others [26].”
- Non-local Neural Networks (2017, methodology): “A natural solution to video classification is to combine the success of CNNs for images and RNNs for sequences [55, 11].”
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, background): “Automatic image captioning has a long history (Ho-dosh et al., 2013; Donahue et al., 2014; Karpa-thy and Fei-Fei, 2015; Kiros et al., 2015).”
- Learning with Latent Language (2017, neutral): “In the running example, this proposal distribution is essentially an image captioning model (Donahue et al., 2015).”
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, background): “Among those in the second category, a common theme has been to use recurrent neural networks to produce novel captions (Kiros et al. 2014; Mao et al. 2014; Karpathy and Fei-Fei 2015;Vinyals et al. 2015;Chen andLawrenceZitnick 2015; Donahue et al. 2015; Fang et al. 2015).”
- Recurrent Neural Networks for Semantic Instance Segmentation (2017, background): “video action classification [13, 43], image captioning [40, 21] or visual question answering [2]).”
- Order Matters: Sequence to sequence for sets (2015, neutral): “For example, image captioning maps from an image to a sentence (Vinyals et al., 2015c; Mao et al., 2015; Donahue et al., 2015), parsing maps from a sentence to a (linearized) parse tree (Vinyals et al., 2015b), and models for computation map from problem statements (e.g. a python program or a set…”
- Generative Adversarial Text to Image Synthesis (2016, background): “However, in the past year, there has been a breakthrough in using recurrent neural network decoders to generate text descriptions conditioned on images (Vinyals et al., 2015; Mao et al., 2015; Karpathy & Li, 2015; Donahue et al., 2015).”
- An Actor-Critic Algorithm for Sequence Prediction (2016, background): “, 2015), caption generation (Kiros et al., 2014; Donahue et al., 2015; Vinyals et al., 2015; Xu et al., 2015; Karpathy and Fei-Fei, 2015), and speech recognition (Chorowski et al.”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “With the advent of deep learning, researchers proposed end-to-end neural networks that encode an image into vector representations and decode a caption word by word [36, 82, 21, 32].”
- Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “, [30,31,32,33,34]) is to increase overall sentence quality.”
- Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “Action understanding remains an intensely studied problem-space, e.g., action recognition (Donahue et al. 2015; Wang et al. 2016), action detection (Singh et al. 2016; Yeung et al. 2016; Shou, Wang, and Chang 2016) and action labeling (Kuehne, Richard, and Gall 2017; Huang, Fei-Fei, and Niebles…”
- Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”