Correcting Robot Plans with Natural Language Feedback

By Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, Dieter Fox. 2022

128 citations recorded by the source corpus.

When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints). Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction. In this paper, we explore natural language as an expressive and flexible tool for robot correction. We describe how to map from natural language sentences to transformations of cost functions. We show that these transformations enable users to correct goals, update robot motions to accommodate additional user preferences, and recover from planning errors. These corrections can be leveraged to get 81% and 93% success rates on tasks where the original planner failed, with either one or two language corrections. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Some works have been trained directly for language-based interaction in robotic tasks [67, 68, 69, 70].”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “UNet + MP [50] trains a U-Net [112, 113] to directly map RGB-D observations to value maps which are then used by a motion planner (MP), thus not having an independent perception module.”
Interactive Language: Talking to Robots in Real Time (2022, background): “For example, [37], [38], [39], and [40] study language corrections, but under the respective simplifying assumptions of hand-defined”
Ignore Previous Prompt: Attack Techniques For Language Models (2022, unknown): “Regarding Q7, we have intentionally modeled our heuristics after human-level content sophistication.”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “Later work explore mapping language corrections to composable cost functions similar to our work by training a prediction model from demonstration and apply trajectory optimization to perform control [25].”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, background): “These systems are often coupled with additional features from pretrained large-scale vision and language models models [11], [14], [15], [16], [37], [38], [39].”
Code as Policies: Language Model Programs for Embodied Control (2022, background): “, model-based [33]–[35], imitation learning [8], [9], [36], [37], or reinforcement learning [38]–[42]).”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Multimodal Large Language Models (MLLMs) have gained significant interest due to their proficiency in analyzing non-textual data like images and point clouds through text analysis [3, 139, 170, 203].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Language costs [98] presents a novel approach to robot correction using natural language for human-in-the-loop robot control systems.”