VIMA: General Robot Manipulation with Multimodal Prompts

By Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, Linxi (Jim) Fan. 2022

578 citations recorded by the source corpus.

Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to $2.9\times$ task success rate given the same training data. With $10\times$ less training data, VIMA still performs $2.7\times$ better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Robotics & Control

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “Among these methods, VIMA (Jiang et al., 2022) explores multimodal prompts similar to PaLM-E.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Other works have incorporated pre-trained language models, often either as an instruction encoder [63, 64, 43, 41, 44, 1, 65] or for high-level planning [66, 19, 17, 67, 68, 69].”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, methodology): “Noticeably, there are other approaches used for improving sample efficiency for imitation learning, such as grounding action on discrete observation space [23, 44, 50, 51].”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “Some researchers [4, 5, 6, 7] explore creating large-scale embodied datasets with simulators, but a significant gap remains between simulation and the real world.”
An Embodied Generalist Agent in 3D World (2023, unknown): “We report the success rate and SPL metrics following Ramrakhya et al. (2022).”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, background): “These systems are often coupled with additional features from pretrained large-scale vision and language models models [11], [14], [15], [16], [37], [38], [39].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “[115] and TIFA [76]), text-to-audio generation (AudioCaps [93]) and action generation (VIMA-Bench [87]).”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “VisuoMotor Attention Agent (VIMA) [152] learns robot manipulation from multi-modal prompts.”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “This idea is directly inspired by successes in gameplaying [4, 34] as well as simulated robot navigation [12], locomotion [24, 15], and manipulation [25] environments.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Similarly, ViMA [157] demonstrates the efﬁcacy of multimodal prompting, interweaving textual and visual tokens, thereby enabling effective learning of robot manipulation through multimodal prompts.”
Vision-Language Foundation Models as Effective Robot Imitators (2023, methodology): “For instance, Jiang et al. (2023) utilizes a pre-trained T5 (Raffel et al., 2020) model to encode the multi-modal prompts, and learn the actions by fine-tuning the T5 model and additionally training an object encoder and attention layers.”
ViNT: A Foundation Model for Visual Navigation (2023, methodology): “We build on the Transformer architecture’s ability to attend to multimodal inputs projected into a shared token space [39, 40].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “VIMA [59] uniﬁes a wide range of robot manipulation tasks with multimodal prompting.”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “The project website is robotics-transformer-x.github.io.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, background): “VIMA (Jiang et al., 2022) enables a multi-modal prompting interface for generalist robot manipulation with an encoder-decoder transformer architecture.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “Presently, SAM-Track [185] for object tracking and VIMA [186] for robot action manipulation showcase potential, implying that these large models can optimize visual recognition systems.”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[19] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi (Jim) Fan.”