LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action

By Dhruv Shah, Blazej Osinski, Brian Ichter, Sergey Levine. 2022

738 citations recorded by the source corpus.

Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings. However, particularly in vision-based settings where specifying goals requires an image, this makes for an unnatural interface. Language provides a more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions. We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories, while still providing a high-level interface to the user. Instead of utilizing a labeled instruction following dataset, we show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data. We instantiate LM-Nav on a real-world mobile robot and demonstrate long-horizon navigation through complex, outdoor environments from natural language instructions. For videos of our experiments, code release, and an interactive Colab notebook that runs in your browser, please check out our project page https://sites.google.com/view/lmnav

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2021a), planning over graphs and maps (Shah et al., 2022; Huang et al., 2022a), visual explanations (Wang et al.”
TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation (2023, background): “Large Language Models (LLMs) have exhibited remarkable proficiency in generating text that closely resembles human language and in performing a wide range of tasks [69], including Natural Language Processing [4, 32, 54], Robotics [10, 43, 56], and Information Retrieval [1, 24, 25, 48].”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Finally, LM-Nav [160] utilizes VLMs to devise actionable plans in an environment.”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Pioneering models like CLIP [139] have shown that training to match images with captions can effectively create image representations from scratch.”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, background): “…al., 2022; Nair et al., 2022; Xiao et al., 2022; Radosavovic et al., 2023; Liu et al., 2022; Seo et al., 2023; Lin et al., 2023; Kumar et al., 2022; Shah et al., 2023; Jing et al., 2023; Parisi et al., 2022; Escontrela et al., 2023; Laskin et al., 2020; Brohan et al., 2023; Liu et al., 2022;…”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “Similarly, LM-Nav [136] considers visual navigation tasks.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “, 2022b) and LM-Nav (Shah et al., 2022) apply LLMs to long-horizon robot planning.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “LM-Nav [119] used a goal-conditioned distance function to infer connections between original observation sets and construct a navigation graph, and extracted landmarks from the instructions through a LLM.”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, methodology): “Different from previous work [8] that uses LLM to extract names from a sentence, our object proposal is much more demanding in four different ways as we will discuss in Sec.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, methodology): “Prior works have studied the use of VLMs for robotics [48, 61, 49, 17, 70, 71, 72], and form part of the inspiration for this work.”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “Existing works use LLMs as a static generator for HLP [12, 13, 22, 30, 33, 40], use LLMs as a ranker that ranks all admissible actions in the current environment [1, 22], or add ground-truth environmental input to the LLM prompt [33].”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, neutral): “Leveraging pre-trained language models for embodied applications is an active area of research, where a large body of works focus on planning and reasoning with language models [9–11, 58, 31, 39, 59–72, 36, 73, 74].”
Visual Language Maps for Robot Navigation (2022, methodology): “• LM-Nav [13] creates a graph where image observations of an environment are stored as nodes while the proximity between images are represented as edges.”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “The recent advent of large language models (LLMs) [1, 2, 3, 4] promises vast transformative potential across multiple sectors, from healthcare [5, 6, 7] and education [8, 9, 10] to robotics [11, 12, 13] and commerce [14].”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “VLMaps [44], LM-Nav [45], CoWs [46], and NLMapSaycan [47] leverage the open-set features from pixel-aligned LSeg [24] models for robot navigation based on language commands.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “Recent success of 2D open-vocabulary segmentation models such as LSeg [39], OpenSeg [18], and OV-Seg[41] has motivated researchers in the field of 3D scene understanding to explore the open vocabulary setting [6, 10, 17, 21, 27, 29, 32, 34, 45, 49, 56, 57].”