Multimodal integration learning of robot behavior using deep neural networks

By Kuniaki Noda, Hiroaki Arie, Yuki Suga, Tetsuya Ogata. 2014

183 citations recorded by the source corpus.

For humans to accurately understand the world around them, multimodal integration is essential because it enhances perceptual precision and reduces ambiguity. Computational models replicating such human ability may contribute to the practical use of robots in daily human living environments; however, primarily because of scalability problems that conventional machine learning algorithms suffer from, sensory-motor information processing in robotic applications has typically been achieved via modal-dependent processes. In this paper, we propose a novel computational framework enabling the integration of sensory-motor time-series data and the self-organization of multimodal fused representations based on a deep learning approach. To evaluate our proposed model, we conducted two behavior-learning experiments utilizing a humanoid robot; the experiments consisted of object manipulation and bell-ringing tasks. From our experimental results, we show that large amounts of sensory-motor information, including raw RGB images, sound spectrums, and joint angles, are directly fused to generate higher-level multimodal representations. Further, we demonstrated that our proposed framework realizes the following three functions: (1) cross-modal memory retrieval utilizing the information complementation capability of the deep autoencoder; (2) noise-robust behavior recognition utilizing the generalization capability of multimodal features; and (3) multimodal causality acquisition and sensory-motor prediction based on the acquired causality. Novel computational framework for sensory-motor integration learning.Cross-modal memory retrieval utilizing a deep autoencoder.Noise-robust behavior recognition utilizing acquired multimodal features.Multimodal causality acquisition and sensory-motor prediction.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Social Psychology

Citations in the corpus

A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments * (2018, background): “, internally generated models of the world) [1].”
Emergence of multimodal action representations from neural network self-organization (2017, background): “Together, these results show that our hierarchical neural architecture accounts for the development of robust multimodal representations from dynamic audiovisual inputs.”
Repeatable Folding Task by Humanoid Robot Worker Using Deep Learning (2017, methodology): “However, their method cannot perform task repeatable and [16], [15] both use a direct teaching method that can only be applied to backdrivable robots.”
Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach (2017, background): “testing platform [16], [33], separate processing of individual modalities [32], and inability to handle temporal information [13].”
SMIL: Multimodal Learning with Severely Missing Modality (2021, background): “Multimodal learning attracts intensive research interest because of broad applications such as intelligent tutoring (Petrovica, Anohina-Naumeca, and Ekenel 2017), robotics (Noda et al. 2014), and healthcare (Frantzidis et al.”
Applications of artificial intelligence for disaster management (2020, background): “2015), handling multiple input data (Ngiam et al. 2011; Noda et al. 2014), and robotic manipulation (Saxena et al.”
Deep learning in robotics: a review of recent research (2017, background): “Noda, Arie, Suga, and Ogata [114] developed a novel deep learning solution involving both unsupervised methods and recurrent models for integrating multi-modal sensorimotor data, including RGB images, sound data, and joint angles.”
A computational model of crossmodal processing for conflict resolution (2017, background): “Artificial systems embedded with multimodal processing capabilities may result in a more robust perceptual experience [12], especially in the case of sensory uncertainty [13].”
A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder (2017, methodology): “It has also been applied for reconstructing time-series data using a sliding time-window [16].”
Behavior recognition for humanoid robots using long short-term memory (2016, methodology): “The behaviors are adapted from the study by Noda et al.(34) Sampling is done by demonstrating the behavior on the NAO robot while recording the encoder values from each joint.”