Real-World Robot Learning with Masked Visual Pre-training

By Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, Trevor Darrell. Conference on Robot Learning, 2022

356 citations recorded by the source corpus.

In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81%). Finally, we train a 307M parameter vision transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/979810ca765695a481c37126103b8ba256ee2192

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “…EmbCLIP [44], and CoW [45]) R3M [39] ResNet-50[16] Time-CL x i : i -th video frame, i j k Sim-Mani : Meta-World, Franka Kitchen, Adroit MVP [41] ViT-BViT-L MAE [46] Ni =1 − log P x i | x ̸ = i x i : image patch Real-Mani (xArm7): pick, reach block, push cube, close fridge VIP [40]…”
Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks (2023, background): “In the robot learning community, there has been an increase in the number of benchmark simulation environments that focus on goal-oriented tasks, notably the pixmc environments [18] and the Franka kitchen environment [6].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “At best, the visual representation might be pretrained on some dataset [65, 79, 91, 100, 115, 117].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “4 faithfully corresponds to the extra bits of information seen during further training), we initialize variants that directly learn from raw pixels with MVP pre-trained ViT (Xiao et al., 2022; Radosavovic et al., 2022).”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, methodology): “, 2021) was pretrained on 400M image-text pairs from the web; MVP (Radosavovic et al., 2022) on 4.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “Some methods aim to learn useful visual representations by masked image modeling (Xiao et al., 2022; Radosavovic et al., 2022; Seo et al., 2023; Karamcheti et al., 2023) and contrastive learning (Nair et al., 2022; Sermanet et al., 2018; Jing et al., 2023; Laskin et al., 2020).”
ViNT: A Foundation Model for Visual Navigation (2023, methodology): “We compare the ViNT backbone to several alternatives, including visual representations trained with supervised learning [2], unsupervised objectives [7, 42, 43], and an embodiment-agnostic navigation policy [18].”
R3M: A Universal Visual Representation for Robot Manipulation (2022, methodology): “Additionally we compare to the MVP model [70], which trains a ViT-B masked auto-encoder on the Ego-soup dataset, which comprises of Ego4D and other egocentric video datasets..”
HumanPlus: Humanoid Shadowing and Imitation from Humans (2024, methodology): “Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “…Code-as-Policy [42], ChatGPT-Robotics [43] Robot Transformers e.g. RT-1 [44], RT-2 [45], RT-X [46], PACT [47], Xiao et al. [48], Radosavovic et al. [49], LATTE [50] R e l e v a n t t o R obo ti c s Perception Open-VocabularyObjectDetection and3Dclassiﬁcation e.g. OWL-ViT [51], GLIP [52], Grounding…”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “…also focus on multi-task or few-shot imitation learning, [25, 27, 30, 34, 46, 50, 88, 102], language-conditioned imitation learning [12, 47, 82, 83], imitation from play data [21, 57, 74, 89], using human videos [16, 24, 29, 60, 69, 80, 84, 96], and using task-specific structures [49, 83, 103].”