SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Task Planning

By Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, I. Reid, Niko Sünderhauf. Conference on Robot Learning, 2023

461 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a 'semantic search' for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an 'iterative replanning' pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page https://sayplan.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3843469976bd895851bfa08c8208350745bf649f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “SayPlan [188] proposes using hierarchical 3-”
The rise and potential of large language model based agents: a survey (2025, unknown): “Furthermore, they could gather feedback from tangible or virtual surroundings, such as cues from task accomplishments or post-action observations, aiding them in revising and reﬁning their plans [72,82,178–180].”
Empowering Biomedical Discovery with AI Agents (2024, unknown): “In embodied AI agents, environmental feedback [144, 155] is captured in textual format, acting as a short-term memory that aids reasoning.”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, background): “These approaches have enabled building real-time systems that can dynamically build up hierarchical 3D scene representations [49]–[51], and more recently shown that various robotics planning tasks can benefit from efficiency and compactness of 3DSGs [52], [53].”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “Recently, vision-language-action (VLA) models have garnered signiﬁcant attention for their ability to extend pre-trained vision-languagemodelstoroboticsusinganext-tokenprediction approach.”
eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents (2026, unknown): “SayPlan [Rana et al., 2023] showed that LLMs can plan over graph-structured spatial representations, and eMEM’s tool interface provides exactly that query substrate.”