ChatGPT for Robotics: Design Principles and Model Abilities

By Sai H. Vemprala, Rogerio Bonatti, Arthur Fender C. Bucker, Ashish Kapoor. IEEE Access, 2023

711 citations recorded by the source corpus.

This paper presents an experimental study regarding the use of OpenAI’s ChatGPT for robotics applications. We outline a strategy that combines design principles for prompt engineering and the creation of a high-level function library which allows ChatGPT to adapt to different robotics tasks, simulators, and form factors. We focus our evaluations on the effectiveness of different prompt engineering techniques and dialog strategies towards the execution of various types of robotics tasks. We explore ChatGPT’s ability to use free-form dialog, parse XML tags, and to synthesize code, in addition to the use of task-specific prompting functions and closed-loop reasoning through dialogues. Our study encompasses a range of tasks within the robotics domain, from basic logical, geometrical, and mathematical reasoning all the way to complex domains such as aerial navigation, manipulation, and embodied agents. We show that ChatGPT can be effective at solving several of such tasks, while allowing users to interact with it primarily via natural language instructions. In addition to these studies, we introduce an open-sourced research tool called PromptCraft, which contains a platform where researchers can collaboratively upload and vote on examples of good prompting schemes for robotics applications, as well as a sample robotics simulator with ChatGPT integration, making it easier for users to get started with using ChatGPT for robotics. Videos and blog: aka.ms/ChatGPT-Robotics PromptCraft, AirSim-ChatGPT code: https://github.com/microsoft/PromptCraft-Robotics

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/0ba581718f294db1d7b3dbc159cc3d3380f74606

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Language to Rewards for Robotic Skill Synthesis (2023, methodology): “1 right): i) a Reward Translator, built upon pre-trained Large Language Models (LLMs) [10], that interacts with and understands user intents and modulates all reward parameters ψ and weights w, and ii) a Motion Controller, based on MuJoCo MPC [21], that takes the generated reward and interactively o”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “Please use simple, clear, structured language to describe the road condition, surroundings, abnormal situation and driving strategy.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Therefore, in this paper we ask: can large pretrained vision-language models be integrated directly into low-level robotic control to boost generalization and enable emergent semantic reasoning?”
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (2023, background): “Tool Learning Recent studies have shed light on the burgeoning capabilities of LLMs in mastering tools and making decisions within complex environments (Qin et al., 2023b; Vemprala et al., 2023; Nakano et al., 2021; Qin et al., 2023a; Shen et al., 2023; Wu et al., 2023; Schick et al., 2023; Hao et a”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, neutral): “The project website is robotics-transformer-x.github.io.”
Text2Motion: from natural language instructions to feasible plans (2023, methodology): “[18] grounds an LLM planner to admissible action sets for task planning, [19, 20] explore the integration of LLMs with PDDL [1], and [21, 22] focuses on task-level replanning with LLMs. Tangential works shift the representation of plans from action sequences to code [5, 23–25] and embed task…”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Inareasrelatedtoautonomousdrivingsuchasembodied AI,incorporatingnaturallanguageasﬁne-grainedknowledge andinstructionstocontrolthevisuomotoragenthasachieved notable progress [151], [152], [153], [154].”
Gorilla: Large Language Model Connected with Massive APIs (2023, neutral): “Moreover, the application of API calls in robotics has been explored to some extent [41, 1].”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “The recent advent of large language models (LLMs) [1, 2, 3, 4, 5] promises vast transformative potential across multiple sectors, from healthcare [6, 7, 8] and education [9, 10, 11] to robotics [12, 13, 14] and commerce [15].”
Tool Learning with Foundation Models (2023, neutral): “To this end, we can construct suitable task-specific prompts either through manual design (Vemprala et al., 2023) or retrievial (Zhou et al.”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “The employment of Large Language Models (LLMs) in the field of robotics remains in the primary stage [55, 4].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “ChatGPT for Robotics [172] take advantage of the programming ability of ChatGPT to facilitate “user on the loop” control, a departure from the conventional “engineer in the loop” methodology.”
OpenAGI: When LLM Meets Domain Experts (2023, methodology): “HuggingGPT [32] integrates the Hugging Face hub with task-specific models around ChatGPT to tackle generalized AI tasks.”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “Image-grounded text generation such as GPT-4 is more interactive with human users and can produce commands to execute codes [28] or control robots [88], as opposed to text-to-image generation which only returns an image.”