Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

By Bowen Baker, Ilge Akkaya, P. Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune. 2022

437 citations recorded by the source corpus.

Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision domains such as robotics, video games, and computer use, publicly available data does not contain the labels required to train behavioral priors in the same way. We extend the internet-scale pretraining paradigm to sequential decision domains through semi-supervised imitation learning wherein agents learn to act by watching online unlabeled videos. Specifically, we show that with a small amount of labeled data we can train an inverse dynamics model accurate enough to label a huge unlabeled source of online data -- here, online videos of people playing Minecraft -- from which we can then train a general behavioral prior. Despite using the native human interface (mouse and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-shot capabilities and that it can be fine-tuned, with both imitation learning and reinforcement learning, to hard-exploration tasks that are impossible to learn from scratch via reinforcement learning. For many tasks our models exhibit human-level performance, and we are the first to report computer agents that can craft diamond tools, which can take proficient humans upwards of 20 minutes (24,000 environment actions) of gameplay to accomplish.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Choose Your Weapon: Survival Strategies for Depressed AI Academics (2023, background): “The recent open AI agent that learned to craft a diamond pickaxe in Minecraft required training of 9 days on 720 V100 GPUs [2]; this amounts to a few hundred thousand dollars for a single experiment.”
A Generalist Agent (2022, methodology): “Finally, while Gato uses data consisting of both observations and corresponding actions, the possibility of using large scale observation-only data to enhance agents has been already studied (Baker et al., 2022).”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, methodology): “…based on three key resources, each corresponding to a specific embodied scenario: a curated dataset of Minecraft videos from the internet for OE (Baker et al., 2022), real-world driving data for AD (Caesar et al., 2020), and real-world robot manipulation videos annotated with text instructions…”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “VPT [72] presents video pretraining in which the agent learns to act by watching unlabeled online videos.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, positive): “After synthesizing the video plan, we can convert the resultant image trajectory to executable actions by a non-causal inverse dynamics model [4, 65, 71], which can be efficiently learned from much fewer data compared with the imitation learning pipeline [2, 9].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “Finally, we follow common practice (Baker et al., 2022) to map predicted action tokens to discretized poses of the robot arm.”
Mastering diverse control tasks through world models (2025, neutral): “Unlike the Voyager environment, the MineRL competition environment requires visual perception and low-level actions for movement and the camera, such as having to jump to climb onto a block or rotate the camera to face a block for mining.”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, methodology): “[8] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, methodology): “Similar work (Baker et al., 2022; Wang et al., 2023) has also adapted LM agents to complex multimodal games such as Minecraft (Guss et al., 2019; Fan et al., 2022).”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, background): “The video data in VPT and VIPER are both in-domain data from task environments.”
Genie: Generative Interactive Environments (2024, methodology): “Note that this procedure is similar to prior works that learn from videos (Baker et al., 2022; Torabi et al., 2018).”
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (2025, positive): “Since the generated videos do not have action labels, we use either latent or inverse dynamics models (IDM) labeled actions (Baker et al., 2022) and train the policy model to treat these pseudo-actions as action labels for a different embodiment.”
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, methodology): “For RL-based methods like VPT [2] and DreamerV3 [7], the goal item(diamond) is hard-coded into the model weights, which means there are no easy way to re-task the trained RL agents for collecting other items in the inference stage.”
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (2023, methodology): “VPT further boosts the success rate to 20% within 20 minutes of play through pre-training on collects ∽ 70k hours human demonstrations and finetuning with human-designed reward function [3].”
Learning Universal Policies via Text-Guided Video Generation (2023, methodology): “While methods such as VPT (Baker et al., 2022) can utilize internet-scale data through learning an inverse dynamics model to label unlabled videos, an inverse dynamics model itself does not support model-based planning or reinforcement learning to further improve learned policies beyond imitation le”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, background): “Most recently, researchers have explored scaling of RL algorithms as a solution to the aforementioned challenges (Baker et al., 2022; Jia et al., 2022; Xu et al., 2023; Kumar et al., 2023; Hafner et al., 2023).”
Tool Learning with Foundation Models (2023, positive): “The supervision from feedback can also enhance the capabilities of an agent trained in a supervised way (Nakano et al., 2021; Baker et al., 2022).”
Language Models can Solve Computer Tasks (2023, methodology): “First, our experiments use LLMs on HTML code, but ideally methods based on multimodal foundation models [13, 41, 2, 35] will be able to take actions based on text, images, audio, and video as input [3, 15, 33, 55].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “VPT [10] is a concurrent work that learns an inverse dynamics model from human contractors to pseudo-label YouTube videos for behavior cloning.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “VPT [68] proposes a video pretraining method that harnesses unlabeled internet data to pretrain a foundational model for the game of Minecraft.”