Inner Monologue: Embodied Reasoning through Planning with Language Models
By Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Peter R. Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar and 7 more. 2022
1,446 citations recorded by the source corpus.
Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Chelsea Finn
- Pieter Abbeel
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Li Fei-Fei
In LLM Agents & Tool Use
- Dieter Fox
- Adrian Weller
- Igor Mordatch
- Brian Ichter
- Stephen Marcus McAleer
- Pierre Baldi
- Geunwoo Kim
- Pieter Abbeel
In Reinforcement Learning
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Ofir Nachum
- Shixiang Shane Gu
- Aviral Kumar
- Igor Mordatch
- Karol Hausman
Citations in the corpus
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “, with multiple modalities (Ahn et al., 2022; Huang et al., 2022; Xu et al., 2022) or special domains such as science and medicine (Lewkowycz et al.”
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “[25] builds upon SayCan to enable closed-loop planning by leveraging environment feedback (from e.”
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “…LLM’s generation and an eligible set of instructions (Huang et al., 2022b), incorporating affordance functions (Ahn et al., 2022), visual feedback (Huang et al., 2022c), generating world models (Nottingham et al., 2023; Zellers et al., 2021a), planning over graphs and maps (Shah et al., 2022;…”
- The alignment problem from a deep learning perspective (2022, methodology): “…addition to standard drivers of progress, AI developers are increasingly using ML systems for accelerating programming [OpenAI, 2023a], and developing new architectures [Elsken et al., 2019], algorithms [Fawzi et al., 2022], training data [Huang et al., 2022a], and chips [Mirhoseini et al., 2021].”
- Code as Policies: Language Model Programs for Embodied Control (2022, methodology): “…shows that large language models (LLMs) pretrained on Internet-scale data [11]–[13] exhibit out-of-the-box capabilities [14]–[16] that can be applied to language-using robots e.g., planning a sequence of steps from natural language instructions [16]–[18] without additional model finetuning.”
- GPT-Driver: Learning to Drive with GPT (2023, background): “5’s capability for detailed numerical reasoning in motion planning.”
- Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “Recently, researchers have shown that the the success of LLMs can be extended to embodied AI domains [32], [33], [42], [194], where “embodied” typically refers to a virtual embodiment in a world simulator, not a physical robot embodiment.”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “These models are capable of solving diverse tasks in zero-shot settings [20] or with the aid of a few examples [49], and they show great potential in planning and decision-making akin to human beings [14].”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Most current robotic systems adopt such a hierarchical framework [9]–[11] because the high-level task planner can leverage models with high capacity while the low-level control policy can focus on speed and precision, similar to hierarchical reinforcement learning.”
- DriveLM: Driving with Graph Visual Question Answering (2023, background): “Embodied Planning with LLMs. Recent work endeavors to leverage the formidable reasoning and generalization capacity of LLMs [20, 36, 68] for embodied AI systems [18, 27, 29, 33, 42, 57, 85].”
- PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change (2022, background): “Additionally, LLMs are also being made to use feedback from users or the environment to improve their planning performance [12, 35, 25].”
- Generative Agents: Interactive Simulacra of Human Behavior (2023, background): “With their ability to generate and decompose action sequences, large language models have also been used in planning robotics tasks [48].”
- Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “More recently, Park et al. (2023) built an agent society using LLMs augmented with memories to simulate human behavior.”
- Reasoning with Language Model is Planning with World Model (2023, background): “Recently, prompting LLMs to do planning directly has gained attention and shown potential (Huang et al., 2022; Singh et al., 2022; Ding et al., 2023).”
- Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, background): “Inner Monologue [9] leveraged environment feedback to improve the planning ability of LLM.”