Adaptive Feature Abstraction for Translating Video to Text

By Yunchen Pu, Martin Renqiang Min, Zhe Gan, Lawrence Carin. 2016

29 citations recorded by the source corpus.

Previous models for video captioning often use the output from a specific layer of a Convolutional Neural Network (CNN) as video features. However, the variable context-dependent semantics in the video may make it more appropriate to adaptively select features from the multiple CNN layers. We propose a new approach for generating adaptive spatiotemporal representations of videos for the captioning task. A novel attention mechanism is developed, that adaptively and sequentially focuses on different layers of CNN features (levels of feature "abstraction"), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on three benchmark datasets: YouTube2Text, M-VAD and MSR-VTT. Along with visualizing the results and how the model works, these experiments quantitatively demonstrate the effectiveness of the proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantics.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Dongjin Song
2. Yao Qin
3. Wei Cheng
4. Haifeng Chen
5. Chunyuan Li
6. Garrison W. Cottrell
7. Guofei Jiang
8. Zhe Gan

Citations in the corpus

• A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction (2017, methodology): “The proposed dual-stage attention-based recurrent neural network (DA-RNN) not only can be used for time series prediction, but also has the potential to serve as a general feature learning tool in computer vision tasks [Pu et al. , 2016; Qin et al. , 2015].”
• Towards Better Representations with Deep/Bayesian Learning (2018, background): “…et al., 2016a; Pu et al., 2016a,b; Chen et al., 2018b; Li et al., 2018a; Pu et al., 2017b; Chen et al., 2018a; Xu et al., 2017; Wang et al., 2017b; Pu et al., 2018b; Yuan et al., 2017, 2018; Chapfuwa et al., 2018), sigmoid belief networks (Gan et al., 2015b,c,a; Zhang et al., 2016a; Henao et al.,…”
• Attention, please! A survey of neural attention models in deep learning (2021, unknown): “2016), image-text data (Tian et al. 2018a; Pu et al. 2018), monocular/RGB-D images (Liu et al.”
• Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Video-and-Language Research With the emergence of large-scale video datasets [6, 1, 29, 11, 58], several videoand-language tasks have been proposed, such as video captioning [21, 56, 62, 18, 33, 16, 47, 59], localizing video segments from natural language queries [19, 3, 8, 37], video reasoning [65]”