Self-organizing neural integration of pose-motion features for human action recognition

By German Ignacio Parisi, Cornelius Weber, Stefan Wermter. 2015

88 citations recorded by the source corpus.

The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Stefan Wermter
2. German Ignacio Parisi
3. Jun Wan
4. Luiza Mici
5. Jun Tani
6. S. Li
7. Cornelius Weber
8. Guo-qing Guo

In Computer Vision and Pattern Recognition

1. German Ignacio Parisi
2. Stefan Wermter
3. Luiza Mici
4. Jun Wan
5. Jun Tani
6. Cornelius Weber
7. S. Li
8. Guo-qing Guo

In Human-Computer Interaction

1. Stefan Wermter
2. German Ignacio Parisi
3. Jun Wan
4. Luiza Mici
5. S. Li
6. Jun Tani
7. Cornelius Weber
8. Guo-qing Guo

Citations in the corpus

• Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, methodology): “Such compression of the feature vectors is desirable since the Gamma-GWR uses the Euclidean distance as a metric to compute the BMUs, which becomes weakly discriminant when the data are very high-dimensional or sparse (Parisi et al., 2015).”
• An Incremental Self-Organizing Architecture for Sensorimotor Learning and Prediction (2017, background): “in time memory or recurrent connections [16], [17], [41], [42].”
• Lifelong learning of human actions with deep neural network self-organization (2017, methodology): “Different from previous models in which action representations are learned from handcrafted 3D features (Parisi et al., 2016, 2015), we use a hierarchy of recurrent networks to learn prototype action featureswith increasingly large spatiotemporal receptive fields.”
• A self-organizing neural network architecture for learning human-object interactions (2017, methodology): “We adopt hierarchical GWR learning [18] for the data processing and subsequent action-object integration.”
• Emergence of multimodal action representations from neural network self-organization (2017, methodology): “For this purpose, we extended our recently proposed spatiotemporal hierarchy for the integration of pose-motion action cues (Parisi et al., 2015) to include an associative network layer where action–word mappings develop from co-occurring audiovisual inputs using asymmetric inter-layer connectivity.”
• Lifelong Learning of Action Representations with Deep Neural Self-Organization (2017, unknown): “In line with previous work on the self-organizing neural integration of action representations (Parisi, Weber, and Wermter 2015), the proposed deep architecture is composed of two distinct processing streams for pose and motion features, and their subsequent integration in the superior temporal…”
• A Human Activity Recognition System Using Skeleton Data from RGBD Sensors (2016, unknown): “Each training/testing procedure is repeated [43] 67.9 55.5 Gaglio et al. [29] 77.3 76.7 Proposed (𝑃 = 15) 87.9 86.7 Faria et al. [47] 91.1 91.9 Parisi et al. [48] 91.9 90.2 Proposed (𝑃 = 7) 92.7 91.5 Shan and Akella [46] 93.8 94 .”
• Explore Efficient Local Features from RGB-D Data for One-Shot Learning Gesture Recognition (2016, methodology): “That is because: (1) we just used RGB-D data while other state-of-the-art methods [61], [62], [63] used skeleton information which is more easier to locate the human body.”
• Human motion assessment in real time using recurrent self-organization (2016, unknown): “, [3]).”
• Multi-modal integration of dynamic audiovisual patterns for an interactive reinforcement learning scenario (2016, methodology): “The GWR training algorithm for attaching labels to neural activation trajectories was discussed in [21].”
• Emotion Recognition from Body Expressions with a Neural Network Architecture (2017, methodology): “The use of a hierarchy of GWR networks for learning actions from body motion patterns was proposed by Parisi et al. [19].”
• Recognition of Transitive Actions with Hierarchical Neural Network Learning (2016, methodology): “This approach has been shown to be more suitable than SOM for learning a set of actions from features based on noisy tracked skeletons [10].”