Vision-Language Foundation Models as Effective Robot Imitators

By Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chi-Hou Cheang, Ya Jing, Weinan Zhang, Huaping Liu and 2 more. International Conference on Learning Representations, 2023

418 citations recorded by the source corpus.

Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.

View Paper: https://www.semanticscholar.org/paper/857efca0d4cbece97da4b904de1d506c2af25f9c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “In contrast, some prior works [10, 39, 46] leverage pretrained VLMs to generate robotic actions by tapping into world knowledge from large-scale vision-language datasets.”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “…vision-language models (VLMs) to directly predict low-level robotic control actions, producing “vision-language-action” models (VLAs) [4, 33, 23, 22, 7, 15, 8, 50, 55, 51, 3, 2], which have demonstrated effective generalization to out-of-distribution test conditions and unseen semantic…”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “By understanding the world through interaction and learning its workings via abductive reasoning, we can further enhance the adaptability, decision reliability, and generalization capabilities of embodied agents in complex real-world environments.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “A number of recent works have explored similar recipes to ours and directly fine-tuned large pretrained VLMs for predicting robot actions [1, 7, 17, 18, 72–74].”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, unknown): “Among them, the development of large-scale Vision-Language-Action (VLA) models [8, 30, 32] are particularly promising, which em-powers robots to perform complex tasks guided by natural language instructions and potentially manage objects or environments that deviate from the training distribution.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “RoboFlamingo [121] adapts the existing VLM, Flamingo [32], [149], to a robot policy by attaching an LSTM-based policy head to the VLM.”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, unknown): “Recently, several studies [34, 7, 30, 31, 71, 33, 70, 51] propose to build generalist robot policies by extending pre-trained VLMs with ability to robot action generation.”