Lifelong learning of human actions with deep neural network self-organization

By German Ignacio Parisi, Jun Tani, Cornelius Weber, Stefan Wermter. 2017

128 citations recorded by the source corpus.

Lifelong learning is fundamental in autonomous robotics for the acquisition and fine-tuning of knowledge through experience. However, conventional deep neural models for action recognition from videos do not account for lifelong learning but rather learn a batch of training data with a predefined number of action classes and samples. Thus, there is the need to develop learning systems with the ability to incrementally process available perceptual cues and to adapt their responses over time. We propose a self-organizing neural architecture for incrementally learning to classify human actions from video sequences. The architecture comprises growing self-organizing networks equipped with recurrent neurons for processing time-varying patterns. We use a set of hierarchically arranged recurrent networks for the unsupervised learning of action representations with increasingly large spatiotemporal receptive fields. Lifelong learning is achieved in terms of prediction-driven neural dynamics in which the growth and the adaptation of the recurrent networks are driven by their capability to reconstruct temporally ordered input sequences. Experimental results on a classification task using two action benchmark datasets show that our model is competitive with state-of-the-art methods for batch learning also when a significant number of sample labels are missing or corrupted during training sessions. Additional experiments show the ability of our model to adapt to non-stationary input avoiding catastrophic interference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

On the role of neurogenesis in overcoming catastrophic forgetting (2018, background): “Here, we focus on growing recurrent self-organizing networks that learn the spatiotemporal structure of the input in an unsupervised fashion [8].”
Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, methodology): “To enable memory replay in the absence of external sensory input, we extend the Gamma-GWR model by implementing temporal connections that learn trajectories of neural activity in the temporal domain.”
A self-organizing neural network architecture for learning human-object interactions (2017, background): “, with the use of deep neural network self-organization [67].”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, background): “Specifically, there have been many ventures into but not limited to single-label text classification [19], question answering [19], language instruction and translation [50], object detection [77,53], captioning [64] and even video representation learning [67,68].”
Continual Learning for Affective Robotics: Why, What and How? (2020, methodology): “Other CLS-based approaches employ selforganising neural models for encoding sensory experiences in the memory [31], [37], [42].”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “consider a different lifelong learning scenario in [552].”
Lifelong Learning and Personalization in Long-Term Human-Robot Interaction (LEAP-HRI) (2021, background): “Lifelong learning models aim to reflect a number of properties of biological systems and their ability to adapt over time with dedicated mechanisms that facilitate learning from novel input while protecting consolidated memories (e.g., [17, 19, 20]).”
Online Continual Learning on Sequences (2020, background): “[62, 63, 72]) For large-scale datasets, these models may have scalability issues and require additional modulatory mechanisms that control their growth over time.”