Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models

By Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, Sergey Levine. International Conference on Learning Representations, 2023

318 citations recorded by the source corpus.

If generalist robots are to operate in truly unstructured environments, they need to be able to recognize and reason about novel objects and scenarios. Such objects and scenarios might not be present in the robot's own training data. We propose SuSIE, a method that leverages an image-editing diffusion model to act as a high-level planner by proposing intermediate subgoals that a low-level controller can accomplish. Specifically, we finetune InstructPix2Pix on video data, consisting of both human videos and robot rollouts, such that it outputs hypothetical future"subgoal"observations given the robot's current observation and a language command. We also use the robot data to train a low-level goal-conditioned policy to act as the aforementioned low-level controller. We find that the high-level subgoal predictions can utilize Internet-scale pretraining and visual understanding to guide the low-level goal-conditioned policy, achieving significantly better generalization and precision than conventional language-conditioned policies. We achieve state-of-the-art results on the CALVIN benchmark, and also demonstrate robust generalization on real-world manipulation tasks, beating strong baselines that have access to privileged information or that utilize orders of magnitude more compute and training data. The project website can be found at http://rail-berkeley.github.io/susie .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d6bd400073090b88ea535a6166ca9c164b8015b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, unknown): “For image generation, we compare 3D-VLA with three types of image generation methods: 1) image-editing methods Instruct-P2P (Brooks et al., 2023); 2) goal image/video generation methods SuSIE (Black et al., 2023); 3) LLMs with image generation ability NeXT-GPT (Wu et al., 2023).”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “After synthesizing the video plan, we can convert the resultant image trajectory to executable actions by a non-causal inverse dynamics model [4, 65, 71], which can be efficiently learned from much fewer data compared with the imitation learning pipeline [2, 9].”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, methodology): “For the video-to-action model, we use a pre-trained Susie policy without extra fine-tuning.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, unknown): “While prior work has explored subgoal generation and goal-conditioned imitation learning [2, 9, 41, 49], to the best of our knowledge, our approach is the first to integrate these concepts with VLAs as intermediate chain-of-thought reasoning steps.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “These models run efﬁciently, Vision-language models for robot learning: Robot learning [9], [18], [19], [20], [21] is an crucial topic in the robotics.”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, background): “…in response to images and prompts [31, 32, 33, 34, 35] have resulted in a large number of works that explore the integration of such models into robot learning pipelines, e.g., to generate goals [36], to provide reward signals [37, 38, 39], or to learn visual state representations [40, 41, 42].”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “Recent works [2, 38, 80] explore the use of large-scale pre-training to boost their performance.”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “Other works incorporate goal image [9] or video [24, 82, 14] prediction tasks to generate valid, executable plans conditioned on language instructions, with these visual cues subsequently guiding the policy in action generation.”
AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems (2025, unknown): “Several prior arts use web-scale video only to facilitate policy learning given the limited scale of action-labeled robot datasets [21], [22], [23].”