Reward Design with Language Models

By Minae Kwon, Sang Michael Xie, Kalesha Bullard, Dorsa Sadigh. International Conference on Learning Representations, 2023

323 citations recorded by the source corpus.

Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/d318e0169f649656c71f02a1f84194a734fe1962

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Preference Ranking Optimization for Human Alignment (2023, background): “Secondly, it can engage in self-bootstrapping to rectify the model’s inadequate responses (Kwon et al., 2023).”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “[35] It considers a universal reward design by prompting the LLM as a proxy reward function, where the user provides a textual prompt with a few examples or a description of the desired behavior.”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, unknown): “…2023) to generate grounded and structured programmatic output for decision making and robotics problems (Liang et al., 2023; Singh et al., 2023; Wang et al., 2023b; Huang et al., 2023; Wang et al., 2023a; Liu et al., 2023; Silver et al., 2023; Ding et al., 2023; Lin et al., 2023; Xie et al., 2023).”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, unknown): “Recent works have also explored related techniques for generating rewards from LLMs (Roit et al., 2023; Kwon et al., 2022; Yang et al., 2023).”
Language to Rewards for Robotic Skill Synthesis (2023, background): “The idea of grounding language to reward has been explored by prior work for extracting user preferences and task knowledge [22, 23, 24, 25, 26, 27].”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “In particular, language-driven rewards can be used to guide RL agents by providing shaped rewards [16].”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, background): “Recent works have also explored related techniques for generating rewards from LLMs (Roit et al., 2023; Kwon et al., 2022; Yang et al., 2023).”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “A related line of works has also explored using LLMs for reward specification in the context of reward design [75] and exploration in reinforcement learning [76–79], and human preference learning [80].”