Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

By Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Fei-Fei Li, Animesh Garg, Jeannette Bohg. 2019

283 citations recorded by the source corpus.

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is nontrivial to manually design a robot controller that combines these modalities, which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to train directly on real robots due to sample complexity. In this article, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Andrew Zisserman
2. Cordelia Schmid
3. João Carreira
4. Quoc V. Le
5. Mohit Bansal
6. Trevor Darrell
7. Christoph Feichtenhofer
8. Jitendra Malik

In Cognitive Neuroscience

1. Andrew Zisserman
2. Oriol Vinyals
3. Andrew Brock
4. João Carreira
5. Derya Soydaner

In Control and Systems Engineering

1. Andrew Zisserman
2. Oriol Vinyals
3. Andrew Jaegle
4. João Carreira
5. Yuanzhi Li
6. Neel Guha

In Reinforcement Learning

1. Andrew Zisserman
2. Oriol Vinyals
3. Andrew Brock
4. Andrew Jaegle
5. João Carreira

In Multimodal Learning

1. Andrew Zisserman
2. Oriol Vinyals
3. Andrew Brock
4. Kristen Grauman
5. Ruohan Gao
6. Lorenzo Torresani
7. Du Tran
8. Bruno Korbar

Citations in the corpus

• Perceiver: General Perception with Iterative Attention (2021, methodology): “Currently, separate feature extractors are used for each modality (Kaiser et al., 2017; Arandjelovic & Zisserman, 2018; Wang et al., 2020b; Chen et al., 2020b; Alayrac et al., 2020; Lee et al., 2020; Xiao et al., 2020) – it is not sensible to concatenate an audio spectrogram or a raw audio sequence ”
• Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “Thisapproachcansignificantlyenhancetheaccuracy andfeasibilityoftaskplanning,betteraddressingthechallenges ofreal-worldenvironments.”