Text2Motion: from natural language instructions to feasible plans

By Kevin Qinghong Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, Jeannette Bohg. Autonomous Robots, 2023

408 citations recorded by the source corpus.

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills. Qualitative results are made available at https://sites.google.com/stanford.edu/text2motion.

Subfields: Artificial Intelligence, Control and Systems Engineering, Ocean Engineering

View Paper: https://www.semanticscholar.org/paper/8f2d4758e6d525509ae36bb30224dc9259027e6b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Ocean Engineering

Citations in the corpus

Reasoning with Language Model is Planning with World Model (2023, background): “Recently, prompting LLMs to do planning direcly has gained attention and shown potential [24, 23, 53, 13, 35].”
TidyBot: Personalized Robot Assistance with Large Language Models (2023, methodology): “This can be done with value functions (Brohan et al., 2022; Lin et al., 2023), semantic translation into admissible actions (Huang et al., 2022), scene description as context (Zeng et al., 2022; Mees et al., 2022; Chen et al., 2022; Singh et al., 2022), feedback (Huang et al., 2022; Yao et al.,…”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “This work considers robot policy learning from demonstrations for contact-rich object manipulation tasks.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, background): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, neutral): “…2023) to generate grounded and structured programmatic output for decision making and robotics problems (Liang et al., 2023; Singh et al., 2023; Wang et al., 2023b; Huang et al., 2023; Wang et al., 2023a; Liu et al., 2023; Silver et al., 2023; Ding et al., 2023; Lin et al., 2023; Xie et al., 2023).”
ExpeL: LLM Agents Are Experiential Learners (2023, methodology): “On the other hand, prompting methods can augment an LLM with better sequential decision-making planning abilities with only a few in-context examples (Hao et al. 2023; Lin et al. 2023b; Sun et al. 2023).”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, methodology): “SayCan [1] and Text2Motion [29] employ an LLM as a heuristic by utilizing it to score high-level actions, followed by a low-level planner that grounds these actions to determine the executability in the physical world.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “TWM [75] also investigates the application of Transformers in building world models.”
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (2023, background): “There are also some concurrent works on planning with LLMs [36, 26, 23, 31, 43].”