Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection

By Mohammadreza Zolfaghari, Gabriel L. Oliveira, Nima Sedaghat, Thomas Brox. 2017

225 citations recorded by the source corpus.

General human action recognition requires understanding of various visual cues. In this paper, we propose a network architecture that computes and integrates the most important visual cues for action recognition: pose, motion, and the raw images. For the integration, we introduce a Markov chain model which adds cues successively. The resulting approach is efficient and applicable to action classification as well as to spatial and temporal action localization. The two contributions clearly improve the performance over respective baselines. The overall approach achieves state-of-the-art action classification performance on HMDB51, J-HMDB and NTU RGB+D datasets. Moreover, it yields state-of-the-art spatio-temporal action localization results on UCF101 and J-HMDB.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, neutral): “(Zolfaghari et al., 2017) RGB Chained Multi-stream Markov chain 80.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classification methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “Very recently, multi-frame approaches have emerged: Tubelets [41] jointly estimate localization and classification over several frames, T-CNN [16] use 3D convolutions to estimate short tubes, micro-tubes rely on two successive frames [33] and pose-guided 3D convolutions add pose to a two-stream appr”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “Zolfaghari [313] designed a three-stream 3DCNN to process pose, motion, and the raw RGB images.”
A flexible model for training action localization with varying levels of supervision (2018, unknown): “The most recent methods [13, 15, 18, 35, 49] operate on multiple frames and leverage temporal information to determine actions.”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, background): “Most recent works on video classification are based on deep learning [6,7,8,9,10].”
Attentional Pooling for Action Recognition (2017, background): “More recent work [58] adds pose as an additional stream in chained multi-stream fashion and shows significant improvements.”
PoTion: Pose MoTion Representation for Action Recognition (2018, methodology): “Our multi-stream performance is also significantly higher than [48] when used in combination with either TSN or I3D.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “…[8, 52, 64, 60, 20], temporal action localization [66, 45, 65, 12, 70], and spatio-temporal action detection [19, 42, 41, 48, 4, 44, 62, 30, 74, 21, 26, 43] rely on pre-training a model on a large-scale dataset such as ImageNet or Kinetics and ﬁnetuning the pre-trained model on the target…”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, background): “Since then, many works on video understanding follow the same multi-stream 2D CNN design, and have made improvements in terms of backbone architecture [10, 52, 30, 12], fusion of the streams [11, 8, 9, 58] and exploiting richer temporal structures [6, 54, 53].”