VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training

By Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, Amy Zhang. International Conference on Learning Representations, 2022

495 citations recorded by the source corpus.

Reward and representation learning are two long-standing challenges for learning an expanding set of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question. We introduce $\textbf{V}$alue-$\textbf{I}$mplicit $\textbf{P}$re-training (VIP), a self-supervised pre-trained visual representation capable of generating dense and smooth reward functions for unseen robotic tasks. VIP casts representation learning from human videos as an offline goal-conditioned reinforcement learning problem and derives a self-supervised dual goal-conditioned value-function objective that does not depend on actions, enabling pre-training on unlabeled human videos. Theoretically, VIP can be understood as a novel implicit time contrastive objective that generates a temporally smooth embedding, enabling the value function to be implicitly defined via the embedding distance, which can then be used to construct the reward for any goal-image specified downstream task. Trained on large-scale Ego4D human videos and without any fine-tuning on in-domain, task-specific data, VIP's frozen representation can provide dense visual reward for an extensive set of simulated and $\textbf{real-robot}$ tasks, enabling diverse reward-based visual control methods and significantly outperforming all prior pre-trained representations. Notably, VIP can enable simple, $\textbf{few-shot}$ offline RL on a suite of real-world robot tasks with as few as 20 trajectories.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Neurology

View Paper: https://www.semanticscholar.org/paper/3fbe2e8413df0207c26ff393c9aaa8488e3ca4c3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Neurology

Citations in the corpus

TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, background): “Such metrics already exist in the wild, e.g. , success labels, human preferences or interventions (Ouyang et al., 2022), or the embedding distance between a current observation and a goal (Eysen-bach et al., 2022; Ma et al., 2022) within a pre-existing learned representation.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, unknown): “…Jing et al., 2023; Parisi et al., 2022; Escontrela et al., 2023; Laskin et al., 2020; Brohan et al., 2023; Liu et al., 2022; Thomas et al., 2023; Schwarzer et al., 2021; Sun et al., 2023; Yen-Chen et al., 2020; Mendonca et al., 2023; Ma et al., 2022; Bhateja et al., 2023; Karamcheti et al., 2023).”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, unknown): “Recent works have also proposed using pretrained foundation models that can produce reward functions for new tasks (Ma et al., 2022; 2023; Fan et al., 2022; Du et al., 2023a; Karamcheti et al., 2023; Du et al., 2023b; Kwon et al., 2023).”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, methodology): “Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents [3–9].”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, background): “Utilizing passive human demonstrations, previous works learn task cost functions [37, 8, 1, 21], affordance functions [2], dense object descriptors [40, 24, 39], action correspondences [33, 28], and pre-trained visual representations [23, 48].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “2) Pre-training for better representations: MaskViT (Gupta et al., 2022b), R3M (Nair et al., 2022), VIP (Ma et al., 2022), and VC-1 (Majumdar et al., 2023) pre-train general visual representations for robotic perception.”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “The model can also interpret relations between objects to determine which object to pick and where to place it, despite no such relations being provided in the robot demonstrations.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “This objective enriches the semantic relevance embedded in the PVRs. VIP [40] also capitalizes on video temporal relationship.”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “VIP [31], learns general-purpose reward functions for goal-conditioned robotic manipulation using the Ego4D dataset.”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “Similar to R3M, Value-Implicit Pretraining (VIP) [36] employs time-contrastive learning to capture temporal dependencies in videos, but it does not require video-language alignment.”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, methodology): “…RL innovations have been demonstrated in pick-and-place problems, addressing challenges such as multi-task learning (114, 115, 125), sample efficiency (54), defining and computing reward (120, 121), resetting the environment (117), and utilizing human demonstrations or offline data (122, 116, 124).”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “The learned cost functions are then used by reinforcement learning [102, 99, 107], imitation learning [97, 96], or trajectory optimization [95, 35] to generate robot actions.”