Hidden Two-Stream Convolutional Networks for Action Recognition

By Yi Zhu, Zhenzhong Lan, Shawn D. Newsam, Alexander G. Hauptmann. 2017

310 citations recorded by the source corpus.

Analyzing videos of human actions involves understanding the temporal relationships among video frames. State-of-the-art action recognition approaches rely on traditional optical flow estimation methods to pre-compute motion information for CNNs. Such a two-stage approach is computationally expensive, storage demanding, and not end-to-end trainable. In this paper, we present a novel CNN architecture that implicitly captures motion information between adjacent frames. We name our approach hidden two-stream CNNs because it only takes raw video frames as input and directly predicts action classes without explicitly computing optical flow. Our end-to-end approach is 10x faster than its two-stage baseline. Experimental results on four challenging action recognition datasets: UCF101, HMDB51, THUMOS14 and ActivityNet v1.2 show that our approach significantly outperforms the previous best real-time approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “We can use CNN to extract frame-based features and then conduct temporal fusion [64], or adopt 3D filter and 3D pooling layers to learn motion features [130], or use optical/scene flow to extract motion information [116, 151], or encode the video into images [6, 150, 155], or use RNN/LSTM to model t”
Progressive Teacher-Student Learning for Early Action Prediction (2019, background): “Recently, with the rise of deep learning, many deep learning based methods, including 3D CNN [39] [5] [12] [40] [13] and two stream CNN [36] [8] [46] etc.”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “[42] introduce a two-stream CNN structure that spatial and temporal information is learned on different streams respectively.”
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition (2019, background): “Some works try to jointly optimize optical flow estimation and the classification network [36, 5], or implicitly model optical flow in RGB network [24].”
Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks (2016, methodology): “Unlike the two-stream network with action classification [34], its accuracy for gesture recognition is not improved by combining the spatial and temporal streams.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “To enhance the HAR performance, several other works [108], [109], [110], [111], [112] have investigated 3D CNN”
MARS: Motion-Augmented RGB Stream for Action Recognition (2019, background): “One alternative to this loss, which requires ground-truth optical ﬂow, is using an unsupervised loss to train on action datasets where this ground-truth is unavailable [49].”
A Comprehensive Survey of Vision-Based Human Action Recognition Methods (2019, background): “[96] attempted to integrate the calculation of the optical flow sequence into the network, and merge the original image and optical flow sequence to represent human action.”
A Unified Framework for Multi-Modal Isolated Gesture Recognition (2018, methodology): “To solve the aforementioned issue, we propose a novel two-stream convolutional network (2SCVN) based on the idea of consensus voting adapted from Simonyan and Zisserman (2014a), Wang et al. (2016), and Feichtenhofer et al. (2016b).”
Density-Aware Single Image De-raining Using a Multi-stream Dense Network (2018, background): “leveraged in various applications such as semantic segmentation [45], face-alignment [22], visual tracking [18] crowdcounting [30], single image super-resolution[43], face antiSpoofing [1], action recognition [48], depth estimation [5], single image dehazing [24, 42, 40] and also in single image de-”