Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

By Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, Jeannette Bohg. 2018

456 citations recorded by the source corpus.

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. We present results in simulation and on a real robot.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Revisiting Self-Supervised Visual Representation Learning (2019, background): “com/google/revisiting-self-supervised In robotics, both the result of interacting with the world, and the fact that multiple perception modalities simultaneously get sensory inputs are strong signals which can be exploited to create self-supervised tasks [21, 41, 26, 10].”
i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning (2021, background): “, 2018), robotics (Sermanet et al., 2018; Lee et al., 2019), speech recognition (Ravanelli et al.”
DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation (2020, background): “community, and a large body of literature studies how to integrate touch to improve perception and manipulation [3]–[6].”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, methodology): “Many self-supervised tasks proposed in previous works can be used [9, 11, 12, 13, 14].”
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks (2019, background): “), the generation of reference signals by g(·) can be addressed in multiple ways, ranging from hand-tuned state machines [15, 41], trajectory optimization [14, 45], imitation learning [5, 12, 19, 38], or reinforcement learning (RL) [9, 23].”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “robotic applications, including autonomous driving [5, 41], navigation [15], and manipulation [33, 36, 2].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Neural networks are also highly effective at combining data from multiple sensor modalities or information sources [119, 233, 236, 370, 369, 240, 173].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “On-policy policy gradient algorithms have been used to learn tasks such as peg insertion (Lee et al. 2019), targeted throwing Ghadirzadeh et al. (2017), and dexterous manipulation (Zhu et al. 2019) directly on real-world hardware, and can be further accelerated with example demonstrations (Zhu et…”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, background): “There exist some works (Lee et al. 2019; Falco et al. 2019; Hogan et al. 2020) which already involve the tactile data to assist the robotic grasping tasks.”