ViNT: A Foundation Model for Visual Navigation

By Dhruv Shah, Ajay Kumar Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, Sergey Levine. Conference on Robot Learning, 2023

346 citations recorded by the source corpus.

General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.

View Paper: https://www.semanticscholar.org/paper/d77e806cd177a162fd20445ed6df566e08d58ced

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “To overcome variability in robotic settings and improve generalization, another solution as ViNT [137] presents is to train foundation models on diverse robotic data across various embodiments.”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, methodology): “Masked ViNT: We integrate our goal masking with the ViNT policy [3] to flexibly condition on the observation context c t .”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, unknown): “The success of transformer models [58] in natural language processing and computer vision motivated a number of recent works that collected large-scale demonstration datasets and trained transformer-based policies on them [2, 16, 25, 38, 39, 42, 49, 51, 65, 67].”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, unknown): “More recently, large-scale datasets collected across many domains [41, 68, 63, 67, 14, 49] have been shown to enable generalization of simple but end-to-end learned tasks to new environments [33, 31, 67, 69, 26, 49, 28, 64].”
Octo: An Open-Source Generalist Robot Policy (2024, methodology): “This is in contrast to prior architectures [10, 81], where adding or removing an image input or changing the task specification would require re-initializing or re-training large components of the pre-trained model.”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Prior work has provided initial demonstrations of X-embodiment training [27] and transfer [25, 29, 32] with transformer models.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “(1) As a generalizable predictive model, Vista could be utilized as a forward dynamics model [13, 26] to simulate short-term dynamics and assist long-horizon planning tasks like visual navigation [106].”