Masked Visual Pre-training for Motor Control

By Tete Xiao, Ilija Radosavovic, Trevor Darrell, Jitendra Malik. arXiv.org, 2022

322 citations recorded by the source corpus.

This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels. We first train the visual representations by masked modeling of natural images. We then freeze the visual encoder and train neural network controllers on top with reinforcement learning. We do not perform any task-specific fine-tuning of the encoder; the same visual representations are used for all motor control tasks. To the best of our knowledge, this is the first self-supervised model to exploit real-world images at scale for motor control. To accelerate progress in learning from pixels, we contribute a benchmark suite of hand-designed tasks varying in movements, scenes, and robots. Without relying on labels, state-estimation, or expert demonstrations, we consistently outperform supervised encoders by up to 80% absolute success rate, sometimes even matching the oracle state performance. We also find that in-the-wild images, e.g., from YouTube or Egocentric videos, lead to better visual representations for various manipulation tasks than ImageNet images.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/523acd658742fb9c978e3f7638c09d7ce78af719

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, background): “Utilizing passive human demonstrations, previous works learn task cost functions [37, 8, 1, 21], affordance functions [2], dense object descriptors [40, 24, 39], action correspondences [33, 28], and pre-trained visual representations [23, 48].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “These methods scale very well and achieve strong results on various downstream tasks including motor control [91].”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, methodology): “Closely related to our work are R3M [34] and MVP [57], which use an Internetscale human video dataset Ego4D [18] to pretrain visual representations for subsequent imitation learning.”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “…solution for acquiring a general visual representation for robotic manipulation (Shah & Kumar, 2021; Parisi et al., 2022; Nair et al., 2022; Xiao et al., 2022) This paradigm is favorable to the traditional approach of in-domain representation learning because it does not require any…”
HumanPlus: Humanoid Shadowing and Imitation from Humans (2024, methodology): “Human data has been used extensively for robot learning, including for pre-training visual or intermediate representations or tasks [8, 28, 50, 53, 66, 79, 86, 99] leveraging Internet-scale data [18, 26, 27], and for imitation learning on in-domain human data [3, 12, 43, 47”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “With ViT architecture, Xiao et. al [114] adopted pre-trained visual representations to train various motor control tasks.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Most works focus on pre-trained visual representations that can be used to initialize the encoder of the robot’s camera observations, either via supervised ImageNet classification [53], data augmentation [54, 55, 56, 57] or objectives that are tailored towards robotic control [58, 59, 60, 61, 62].”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, unknown): “Prior methods have explored initializing vision encoders from computer vision datasets [85, 58, 57, 18], or leveraging off-the-shelf task planners [38, 48, 73, 81].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “4 faithfully corresponds to the extra bits of information seen during further training), we initialize variants that directly learn from raw pixels with MVP pre-trained ViT (Xiao et al., 2022; Radosavovic et al., 2022).”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “[38] also explore the use of human interaction data to pre-train visual representations for motor control.”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “We also see that our smallest ViT-S model from [10] outperforms it as well (68.8% vs. 31.3%).”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “…e.g. ProgPrompt [41], Code-as-Policy [42], ChatGPT-Robotics [43] Robot Transformers e.g. RT-1 [44], RT-2 [45], RT-X [46], PACT [47], Xiao et al. [48], Radosavovic et al. [49], LATTE [50] R e l e v a n t t o R obo ti c s Perception Open-VocabularyObjectDetection and3Dclassiﬁcation e.g. OWL-ViT…”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Self-supervised representation learning techniques distill knowledge from external datasets to produce downstream features that are helpful in virtual environments [15, 53].”