Emergence of multimodal action representations from neural network self-organization

By German Ignacio Parisi, Jun Tani, Cornelius Weber, Stefan Wermter. 2017

43 citations recorded by the source corpus.

The integration of multisensory information plays a crucial role in autonomous robotics to forming robust and meaningful representations of the environment. In this work, we investigate how robust multimodal representations can naturally develop in a self-organizing manner from co-occurring multisensory inputs. We propose a hierarchical architecture with growing self-organizing neural networks for learning human actions from audiovisual inputs. The hierarchical processing of visual inputs allows to obtain progressively specialized neurons encoding latent spatiotemporal dynamics of the input, consistent with neurophysiological evidence for increasingly large temporal receptive windows in the human cortex. Associative links to bind unimodal representations are incrementally learned by a semi-supervised algorithm with bidirectional connectivity. Multimodal representations of actions are obtained using the co-activation of action features from video sequences and labels from automatic speech recognition. Experimental results on a dataset of 10 full-body actions show that our system achieves state-of-the-art classification performance without requiring the manual segmentation of training samples, and that congruent visual representations can be retrieved from recognized speech in the absence of visual stimuli. Together, these results show that our hierarchical neural architecture accounts for the development of robust multimodal representations from dynamic audiovisual inputs.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Sensory Systems, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

1. German Ignacio Parisi
2. Stefan Wermter
3. Cornelius Weber
4. Luiza Mici
5. Jun Tani

In Experimental and Cognitive Psychology

1. German Ignacio Parisi
2. Stefan Wermter

In Sensory Systems

1. German Ignacio Parisi
2. Stefan Wermter

In Multimodal Learning

1. Stefan Wermter
2. German Ignacio Parisi

Citations in the corpus

• Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization (2018, background): “Such temporal connections are sequenceselective synaptic links which are incremented between two consecutively activated neurons (Parisi et al., 2016).”
• An Incremental Self-Organizing Architecture for Sensorimotor Learning and Prediction (2017, methodology): “For this purpose, the implementation of bidirectional Hebbian connections would have to be investigated in order to connect the last layer of the proposed architecture with a symbolic layer containing action labels [16], [51] and explore how such symbolic layer can modulate the generation of the mov”
• Multimodal Language Analysis with Recurrent Multistage Fusion (2018, unknown): “Recent advances in cognitive neuroscience have demonstrated the existence of multistage aggregation across human cortical networks and functions (Taylor et al., 2015), particularly during the integration of multisensory information (Parisi et al., 2017).”