Learning Universal Policies via Text-Guided Video Generation

By Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, Pieter Abbeel. Neural Information Processing Systems, 2023

599 citations recorded by the source corpus.

A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents. Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able to naturally and combinatorially generalize to novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available videos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/da2fe6cd385194b0274d04d04ee72e8caf3854d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “One line of work instantiates this by combining text-conditioned video diffusion models with image-goal-conditioned policies to solve manipulation tasks in simulated and real tabletop settings [192].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “UniPi [128] treats the decision-making problem as a text-conditioned video generation problem.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “(2) It is also intriguing to utilize Vista as an implicit driving policy, which is spontaneously acquired through future prediction [1, 25].”
AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems (2025, unknown): “Several prior arts use web-scale video only to facilitate policy learning given the limited scale of action-labeled robot datasets [21], [22], [23].”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, background): “With the advancement of diffusion transformer (Peebles & Xie, 2023) and the extensive utilization of large-scale internet video datasets (Bain et al., 2021; Ebert et al., 2021; Goyal et al., 2017; Grauman et al., 2022), certain Predictive Actionable Video Model (Yang et al., 2023) models, also known”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “Other works incorporate goal image [9] or video [24, 82, 14] prediction tasks to generate valid, executable plans conditioned on language instructions, with these visual cues subsequently guiding the policy in action generation.”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, background): “…of the solution. branches of outcomes are explored during search, is widely used in many planning algorithms (Swiechowski et al., 2021; LaValle, 1998) and reinforcement learning (RL) (Hafner et al., 2019; Du et al., 2023; Wu et al., 2023) algorithms for its good exploration-exploitation trade-off.”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “Diffusion models in robotics Beyond policy representations in imitation learning, diffusion models have been used to model cross-object and object-part arrangements [43, 70, 51, 17, 20] and visual image subgoals [36, 15, 1, 2].”
Learning Interactive Real-World Simulators (2023, methodology): “[49] shows video generation can serve as policies, but the major bottleneck for policy learning often lies in limited access to real-world environments [37].”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, unknown): “Other works have explored the integration of generative modeling into policy learning (Du et al., 2024; Wu et al., 2023a; Zhao et al., 2025; Zhu et al., 2025; Du et al., 2023; Zheng et al., 2025; Rajasegaran et al., 2025).”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “Typically, these prior methods use pretrained models to imagine visual [16, 2] or textual plans [8, 28, 29, 39], which then inform a low-level robot control policy.”