Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks
By Hongsong Wang, Liang Wang. 2017
432 citations recorded by the source corpus.
Recently, skeleton based action recognition gains more popularity due to cost-effective depth sensors coupled with real-time skeleton estimation algorithms. Traditional approaches based on handcrafted features are limited to represent the complexity of motion patterns. Recent methods that use Recurrent Neural Networks (RNN) to handle raw skeletons only focus on the contextual dependency in the temporal domain and neglect the spatial configurations of articulated skeletons. In this paper, we propose a novel two-stream RNN architecture to model both temporal dynamics and spatial configurations for skeleton based action recognition. We explore two different structures for the temporal stream: stacked RNN and hierarchical RNN. Hierarchical RNN is designed according to human body kinematics. We also propose two effective methods to model the spatial structure by converting the spatial graph into a sequence of joints. To improve generalization of our model, we further exploit 3D transformation based data augmentation techniques including rotation and scaling transformation to transform the 3D coordinates of skeletons during training. Experiments on 3D action recognition benchmark datasets show that our method brings a considerable improvement for a variety of actions, i.e., generic actions, interaction activities and gestures.
Subfields: Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Model Architectures
Citations in the corpus
- NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Specifically, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
- Deep Bilinear Learning for RGB-D Action Recognition (2018, background): “Learning time-varying information by data-driven approaches [7, 8, 35] can generalize better to unseen sequences.”
- Human Action Recognition From Various Data Modalities: A Review (2020, background): “In the work of [164], a two-stream RNN structure”
- An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition (2019, methodology): “Recent methods can be grouped into two categories: joint-based [35, 36, 12, 28, 13] and part-based methods [21, 28, 4].”
- Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning (2018, neutral): “A two-stream RNN architecture is proposed to model both temporal dynamics and spatial configurations for skeleton-based action recognition in [28].”
- InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “In the early stages of deep learning-based approaches for skeleton-based action recognition, convolutional neural networks (CNNs) [10, 32, 46] and recurrent neural networks (RNNs) [11,25,31,53] were the standard models to adopt.”
- Pattern Recognition (2018, neutral): “The two-stream network with optical flow in action recognition [46] , skeleton-based action recognition methods [47,48] and the multi-scale context aware network in person re-identification [49] could be helpful to the performance boost.”
- A Semisupervised Recurrent Convolutional Attention Model for Human Activity Recognition (2020, background): “1) Motivations: Intuitively, the motion of different body parts has various contributions to different activities [24].”