Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

By Hongtao Wu, Ya Jing, Chi-Hou Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, Tao Kong. International Conference on Learning Representations, 2023

380 citations recorded by the source corpus.

Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https://GR1-Manipulation.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4443c9a43bff8dcd717e5c75115ec6497af2b953

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, neutral): “Other works incorporate goal image [9] or video [24, 82, 14] prediction tasks to generate valid, executable plans conditioned on language instructions, with these visual cues subsequently guiding the policy in action generation.”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “A group of meth-ods [5, 11, 68] employ large-scale video generative pretraining to enhance visual robot manipulation learning without leveraging pretrained VLMs, and promising results have been demonstrated.”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “Other works have explored the integration of generative modeling into policy learning (Du et al., 2024; Wu et al., 2023a; Zhao et al., 2025; Zhu et al., 2025; Du et al., 2023; Zheng et al., 2025; Rajasegaran et al., 2025).”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, positive): “Observation space and action representation 3D Diffuser Actor is trained on demonstrations of successful trajectories in the form of { ( o 1 , a 1 ) , ( o 2 , a 2 ) , ... } , accompanied with a task language description, similar to previous works [31, 69, 19, 80].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “GR-1 [69] introduces video prediction pretraining tailored for a GPT-style model.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, neutral): “Recent advances in robot learning have demonstrated impressive progress in training policies that can act across diverse tasks and environments [1, 4, 10, 12, 15, 26, 33, 39, 40, 43, 48, 53, 57, 60, 64, 70, 72].”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “We believe our work marks a significant advancement in video generation and opens up new possibilities for applications in robotic learning [13, 89], game rendering [7, 79], streaming video editing [9], and other scenarios that require real-time and long-horizon video generation.”