Language to Rewards for Robotic Skill Synthesis

By Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik and 10 more. Conference on Robot Learning, 2023

422 citations recorded by the source corpus.

Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/94bcf0390d5acb1b92323bd15cc1dc311314122c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, methodology): “For robotic applications, concurrent works explored LLM-based reward generation [82–88], among which Yu et al. [82] use MuJoCo [89] as a high-fidelity physics model for model predictive control.”
A Comprehensive Overview of Large Language Models (2023, positive): “Less is more for alignment (LIMA) [166] uses only 1000 carefully created demonstrations to fine-tune the model and has achieved comparable performance to GPT-4.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, positive): “…works utilize VLMs as intermediate components for perception and control, leveraging their strong semantic understanding and reasoning capabilities to decompose complex tasks [18, 23, 31, 50], detect objects [16, 22], or generate dense rewards [11, 37, 68] or goals [2, 9, 13, 41, 42, 49, 65, 74].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “A body of work on using language feedback for reward inference and shaping might lessen this challenge (Fu et al., 2019; Goyal et al., 2019; Sumers et al., 2021; Zhou and Small, 2021; Lin et al., 2022; Yu et al., 2023), but thus far, these techniques have not been applied to LLMs.”