Two-Stream SR-CNNs for Action Recognition in Videos

By Yifan Wang, Jie Song, Limin Wang, Luc Van Gool, Otmar Hilliges. 2016

105 citations recorded by the source corpus.

Human action is a high-level concept in computer vision research and understanding it may beneﬁt from different semantics, such as human pose, interacting objects, and scene context. In this paper, we explicitly exploit semantic cues with aid of existing human/object detectors for action recognition in videos, and thoroughly study their effect on the recognition performance for different types of actions. Speciﬁcally, we propose a new deep architecture by incorporating human/object detection results into the framework, called two-stream semantic region based CNNs (SR-CNNs). Our proposed architecture not only shares great modeling capacity with the original two-stream CNNs, but also exhibits the ﬂexibility of leveraging semantic cues (e.g. scene, person, object) for action understanding. We perform experiments on the UCF101 dataset and demonstrate its superior performance to the original two-stream CNNs. In addition, we systematically study the effect of incorporating semantic cues on the recognition performance for different types of action classes, and try to provide some insights for building more reasonable action benchmarks and developing better recognition algorithms.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Video Representation Learning Using Discriminative Pooling (2018, background): “For example, in the popular two-stream model [13, 41, 42, 53, 55], the CNNs are”
Knowledge Guided Disambiguation for Large-Scale Scene Classification With Multi-Resolution CNNs (2016, background): “object detection [13], [14], event recognition [15], [16], and action recognition [17], [18].”
Explaining Motion Relevance for Activity Recognition in Video Deep Learning Models (2020, methodology): “Seeking to benefit from motion information provided by optical flow, in [3], the authors used two-stream approach [17, 18, 19] and fused two 3D CNNs, one stream using original video frames and the other using optical flow between the frames, achieving state-of-the-art activity recognition performanc”
Knowledge Integration Networks for Action Recognition (2020, methodology): “Notice that previous works, such as (Wu et al. 2016; Heilbron et al. 2017; Wang et al. 2016b), encode the extra knowledge by directly using the teacher networks whose parameters are ﬁxed, while our proposed three-branch framework with knowledge distillation enables a joint learning of three tasks.”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “Beyond body pose, there is a growing interest in modeling human-human interactions [57, 23, 40] to predict pedestrian trajectories [1, 43, 2], analyze social behavior and group activities [43, 59, 6, 15, 23], and understand human-object interactions [61, 19, 12].”