Cross-View Action Modeling, Learning, and Recognition

By Jiang Wang, Xiaohan Nie, Yin Xia, Ying Wu, Song-Chun Zhu. 2014

631 citations recorded by the source corpus.

Existing methods on video-based action recognition are generally view-dependent, i.e., performing recognition from the same views seen in the training data. We present a novel multiview spatio-temporal and-or graph (MST-AOG) representation for cross-view action recognition, i.e., the recognition is performed on the video from an unknown and unseen view. As a compositional model, MST-AOG compactly represents the hierarchical combinatorial structures of cross-view actions by explicitly modeling the geometry, appearance and motion variations. This paper proposes effective methods to learn the structure and parameters of MST-AOG. The inference based on MST-AOG enables action recognition from novel views. The training of MST-AOG takes advantage of the 3D human skeleton data obtained from Kinect cameras to avoid annotating enormous multi-view video frames, which is error-prone and time-consuming, but the recognition does not need 3D information and is based on 2D video input. A new Multiview Action3D dataset has been created and will be released. Extensive experiments have demonstrated that this new action representation significantly improves the accuracy and robustness for cross-view action recognition on 2D videos.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis (2016, methodology): “Multiview 3D event [43] and Northwestern-UCLA [40] datasets used more than one Kincect cameras at the same time to collect multi-view representations of the same action, and scale up the number of samples.”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “The Northwestern-UCLA [20] and the Multiview 3D Event [19] datasets used more than one depth sensors at the same time to collect multi-view representations of the same action, and scale up the number of samples.”
Space-Time Representation of People Based on 3D Skeletal Data: A Review (2016, methodology): “[106] Cross View BoW Body Dict X X Wei et al.”
Histogram of Oriented Principal Components for Cross-View Action Recognition (2014, background): “Keywords: Spatio-temporal keypoint, pointcloud, view invariance.”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Northwestern-UCLA Multiview Action 3D Dataset (N-UCLA) —”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “Northwestern-UCLA Multiview Action 3D Northwestern-UCLA Multiview Action 3D (Wang et al., 2014)”
Learning a non-linear knowledge transfer model for cross-view action recognition (2015, methodology): “The proposed algorithm was evaluated on two benchmark datasets including the INRIA Xmas Motion Acquisition Sequences (IXMAS) [34] and the Northwestern-UCLA Multiview Action3D (N-UCLA) [33] datasets.”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, methodology): “f the samples of each activity are used for training and the rest for testing. We use the 30-fold cross-validation and show the mean accuracy for each setting [12]. Northwestern-UCLA Dataset (N-UCLA) [40]. This dataset consists of 1494 skeleton sequences. It includes 10 actions performed by 10 diffe”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “Comparative results on NTU RGB+D 60 [42] ( left ), NTU RGB+D 120 [30] ( middle ), and NW-UCLA [55] ( right ).”
Modality Distillation with Multiple Stream Networks for Action Recognition (2018, methodology): “– we propose a new multimodal stream network architecture able to exploit multiple data modalities at training while using only one at test time; – we introduce a new paradigm to learn an hallucination network within a novel two-stream model; – in this context, we have implemented an inter-stream co”
Monocular 3D Human Pose Estimation by Predicting Depth on Joints (2017, methodology): “To better model the kinematic relation of human skeletons, the two recurrent networks have tree-structures similar to the models in [34, 19, 24].”
An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “The evaluation protocol is the same as in [33].”
RPAN: An End-to-End Recurrent Pose-Attention Network for Action Recognition in Videos (2017, neutral): “9 MST [43] 2014 45.”
Learning a Deep Model for Human Action Recognition from Novel Viewpoints (2016, methodology): “Besides the limitations of employing linear transformation functions between views, existing cross-view action recognition methods [15], [18], [20], [21], [22], [23] are either not”
A Comprehensive Survey of Vision-Based Human Action Recognition Methods (2019, neutral): “Hollywood2 [97] √ × × 1707 12 HMDB51 [98] √ × × 6766 51 Olympic Sports [99] √ × × 783 16 UCF50 [100] √ × × 6618 50 UCF101 [101] √ × × 13,320 101 Kinetics [102] √ × × 306,245 400 MSR-Action3D [103] × √ √ 567 20 MSR-Daily Activity [104] √ √ √ 320 16 Northwestern-UCLA [105] √ √ √ 1475 10 UTD-MHAD [106]”
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, methodology): “Northwestern-UCLA dataset [30] is captured by three Kinect cameras simultaneously from multiple viewpoints.”
Learning Clip Representations for Skeleton-Based 3D Action Recognition (2018, methodology): “The proposed method is tested on six skeleton action datasets: NTU RGB+D dataset [9], Northwestern-UCLA Multiview Action 3D Dataset [62], SBU Kinect Interaction Dataset [41], CMU Dataset [63], UTKinect-Action3D Dataset [6] and Berkeley Multimodal Human Action Dataset (Berkeley MHAD) [64].”
Enhanced skeleton visualization for view invariant human action recognition (2017, methodology): “Following [39] , we use samples from the ﬁrst two cameras as training data, and the samples from the third camera as test data.”