Inferring Rewards from Language in Context

By Jessy Lin, Daniel Fried, Daniel Klein, Anca Dragan. 2022

74 citations recorded by the source corpus.

In classic instruction following, language like “I’d like the JetBlue flight” maps to actions (e.g., selecting that flight). However, language also conveys information about a user’s underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight–booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Training Language Models with Language Feedback (2022, background): “…Chaplot et al., 2017; Mahmoudieh et al., 2022; Ouyang et al., 2022, inter alia ), drive exploration (Tam et al., 2022), infer reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and train the model via strong supervision (Andreas et al., 2017; Kaplan…”
Language to Rewards for Robotic Skill Synthesis (2023, background): “The idea of grounding language to reward has been explored by prior work for extracting user preferences and task knowledge [22, 23, 24, 25, 26, 27].”
Training Language Models with Language Feedback at Scale (2023, background): “…for an overview), such as specifying tasks (“instruction following” , inter alia ) driving exploration (Tam et al., 2022), inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and training a model via strong supervision (Andreas et al., 2017; Kaplan…”
Training Language Models with Natural Language Feedback (2022, background): “…Chaplot et al., 2017; Mahmoudieh et al., 2022; Ouyang et al., 2022, inter alia ), drive exploration (Tam et al., 2022), infer reward functions (Lin et al., 2022), and train the model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017) or reward shaping (Goyal et al., 2019).”
Learning from Natural Language Feedback (2024, background): “, 2022) or to infer rewards in a certain context (Lin et al., 2022).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, unknown): “A body of work on using language feedback for reward inference and shaping might lessen this challenge [Fu et al., 2019, Goyal et al., 2019, Sumers et al., 2021, Zhou and Small, 2021, Lin et al., 2022, Yu et al., 2023], but thus far, these techniques have not been applied to LLMs.”
Reward Design with Language Models (2023, background): “Recent work attempts to learn reward functions from language instructions using pragmatic reasoning (Lin et al., 2022).”
The rise and potential of large language model based agents: a survey (2025, unknown): “Large language models exhibit strong natural language interaction abilities like comprehension and generation [17, 112, 113].”