Code as Policies: Language Model Programs for Embodied Control
By Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Peter R. Florence, Andy Zeng. 2022
1,688 citations recorded by the source corpus.
Large language models (LLMs) trained on code-completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g., from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By chaining classic logic structures and referencing third-party libraries (e.g., NumPy, Shapely) to perform arithmetic, LLMs used in this way can write robot policies that (i) exhibit spatial-geometric reasoning, (ii) generalize to new instructions, and (iii) prescribe precise values (e.g., velocities) to ambiguous descriptions (‘faster’) depending on context (i.e., behavioral commonsense). This paper presents Code as Policies: a robot-centric formulation of language model generated programs (LMPs) that can represent reactive policies (e.g., impedance controllers), as well as waypoint-based policies (vision-based pick and place, trajectory-based control), demonstrated across multiple real robot platforms. Central to our approach is prompting hierarchical code-gen (recursively defining undefined functions), which can write more complex code and also improves state-of-the-art to solve 39.8% of problems on the HumanEval [1] benchmark. Code and videos are available at https://code-as-policies.github.io
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, LLM Agents & Tool Use, Code Generation & Program Synthesis, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Chelsea Finn
- Percy Liang
- Pieter Abbeel
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Ming-Hsuan Yang
- Li Fei-Fei
In Control and Systems Engineering
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Li Fei-Fei
- Yuke Zhu
- Pierre Sermanet
- Vikash Kumar
- Nicolas Manfred Otto Heess
In LLM Agents & Tool Use
- Brian Ichter
- Andy Zeng
- Cordelia Schmid
- Chelsea Finn
- Corey Lynch
- Tianmin Shu
- Pierre Sermanet
- Dorsa Sadigh
In Code Generation & Program Synthesis
- Frank F. Xu
- Graham Neubig
- Shuyan Zhou
- Josh Tenenbaum
- Hao Peng
- Prithviraj Ammanabrolu
- Hannaneh Hajishirzi
- Alane Suhr
In Robotics & Control
- Sergey Levine
- Chelsea Finn
- Li Fei-Fei
- Pieter Abbeel
- Corey Lynch
- Yuke Zhu
- Pierre Sermanet
- Nicolas Manfred Otto Heess
Citations in the corpus
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2023), program generation (Liang et al., 2022; Singh et al., 2022), or injecting information into the prompt (Zeng et al.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “Code as policies (CaP) [174] also leverages the code-writing capability of LLMs.”
- A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “Recurrent Neural Networks (RNNs) [148] and Long Short-Term Memory (LSTM) networks [55] were used for various NLP tasks.”
- Eureka: Human-Level Reward Design via Coding Large Language Models (2023, neutral): “…2021; Chen et al., 2021; Rozi`ere et al., 2023) to generate grounded and structured programmatic output for decision making and robotics problems (Liang et al., 2023; Singh et al., 2023; Wang et al., 2023b; Huang et al., 2023; Wang et al., 2023a; Liu et al., 2023; Silver et al., 2023; Ding et…”
- Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, methodology): “[16] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.”
- Foundation models in robotics: Applications, challenges, and the future (2023, positive): “Recently, researchers have shown that the the success of LLMs can be extended to embodied AI domains [32], [33], [42], [194], where “embodied” typically refers to a virtual embodiment in a world simulator, not a physical robot embodiment.”
- Executable Code Actions Elicit Better LLM Agents (2024, background): “These contributions not only extend the work of TaskWeaver but also pave the way for future explorations, offering valuable resources to the open-source community and redefining the potential of LLM agents in practical applications.”
- ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution (2024, neutral): “…(Chen et al., 2023c; Liventsev et al., 2023), enhancing code performance (Madaan et al., 2023a), solving algorithmic competition challenges (Li et al., 2022; Shinn et al., 2023), robotics (Liang et al., 2023; Wang et al., 2023b), and general task solving (Yang et al., 2023b; Zhang et al., 2022).”
- Empowering Biomedical Discovery with AI Agents (2024, neutral): “Continuous training and development of human expertise alongside AI can mitigate the risks of over-reliance on AI.”
- ChatGPT for Robotics: Design Principles and Model Abilities (2023, methodology): “Recent works have also explored the ability of large language models (LLMs) for zero-shot high-level robotics task planning [17], [19], [20].”
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “Typically, these prior methods use pretrained models to imagine visual [16, 2] or textual plans [8, 28, 29, 39], which then inform a low-level robot control policy.”
- Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, background): “Several works [4, 13, 21] have explored to augment LLMs with robot perception and control abilities.”
- DriveLM: Driving with Graph Visual Question Answering (2023, background): “CaP [42] provides a robot-centric formulation of language model generated programs executed on real systems.”
- Language to Rewards for Robotic Skill Synthesis (2023, methodology): “For the Code-as-Policies (CaP) baseline, we design the primitive skills based on common commands available to the robot.”
- Robotic Control via Embodied Chain-of-Thought Reasoning (2024, positive): “A number of works have explored similar techniques in the context of high-level task planning for robotics [48, 49, 50, 51, 52, 53, 54, 55, 56].”
- A Survey of Large Language Models (2023, positive): “Existing work has shown the effectiveness of LLMs in interacting with the external environment and generating accurate action plans [386].”
- Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning (2023, neutral): “, 2022], Code as Policies [Liang et al., 2022] and Inner Monologue [Huang et al.”