Adaptive Feature Abstraction for Translating Video to Language

By Yunchen Pu, Martin Renqiang Min, Zhe Gan, Lawrence Carin. 2016

14 citations recorded by the source corpus.

A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. We consider both "hard" and "soft" attention mechanisms, to adaptively and sequentially focus on different layers of features (levels of feature "abstraction"), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on three benchmark datasets: YouTube2Text, M-VAD and MSR-VTT. Along with visualizing the results and how the model works, these experiments quantitatively demonstrate the effectiveness of the proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Chunyuan Li
4. Lawrence Carin
5. Li Deng
6. Xiaodong He
7. Dinghan Shen
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Jianfeng Gao
3. Chunyuan Li
4. Lawrence Carin
5. Luc Van Gool
6. Dacheng Tao
7. Trevor Darrell
8. Xiaogang Wang

Citations in the corpus

• A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction (2017, methodology): “The proposed dual-stage attention-based recurrent neural network (DA-RNN) not only can be used for time series prediction, but also has the potential to serve as a general feature learning tool in computer vision tasks [Pu et al. , 2016; Qin et al. , 2015].”
• Semantic Compositional Networks for Visual Captioning (2016, background): “Representative works include [7, 9, 10, 20, 23, 24, 29, 48] for image captioning and [10, 46, 47, 56, 3, 35, 11] for video captioning.”
• Video Generation From Text (2017, background): “Previous work on the generative relationship between text and video has focused on producing text captioning from video (Venugopalan et al. 2015; Donahue et al. 2015; Pan et al. 2016; Pu et al. 2017).”