Fusing Multi-Stream Deep Networks for Video Classification

By Zuxuan Wu, Yu-Gang Jiang, Xi Wang, Hao Ye, Xiangyang Xue, Jun Wang. 2015

49 citations recorded by the source corpus.

This paper studies deep network architectures to address the problem of video classification. A multi-stream framework is proposed to fully utilize the rich multimodal information in videos. Specifically, we first train three Convolutional Neural Networks to model spatial, short-term motion and audio clues respectively. Long Short Term Memory networks are then adopted to explore long-term temporal dynamics. With the outputs of the individual streams, we propose a simple and effective fusion method to generate the final predictions, where the optimal fusion weights are learned adaptively for each class, and the learning process is regularized by automatically estimated class relationships. Our contributions are two-fold. First, the proposed multi-stream framework is able to exploit multimodal features that are more comprehensive than those previously attempted. Second, we demonstrate that the adaptive fusion method using the class relationship as a regularizer outperforms traditional alternatives that estimate the weights in a "free" fashion. Our framework produces significantly better results than the state of the arts on two popular benchmarks, 92.2\% on UCF-101 (without using audio) and 84.9\% on Columbia Consumer Videos.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, methodology): “In a similar spirit, Wang et al. [46] presented an adaptive fusion approach, which uses two regularization terms to learn fusion weights.”
Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition (2016, methodology): “While earlier works resorted to unsupervised learning of 3D spatio-temporal features [45], supervised end-to-end learning has recently gained popularity [11,12,13,14,15,16,17,18,19,20,21,22].”
Going deeper into action recognition: A survey (2016, background): “The highest accuracies belong to Wu et al. (2015), Wang et al. (2016), Wang et al. (2015a) and Wang et al. (2015c).”