Learning Rewards from Linguistic Feedback

By Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, T. Griffiths. 2020

69 citations recorded by the source corpus.

We explore unconstrained natural language feedback as a learning signal for artificial agents. Humans use rich and varied language to teach, yet most prior work on interactive learning from language assumes a particular form of input (e.g., commands). We propose a general framework which does not make this assumption, instead using aspect-based sentiment analysis to decompose feedback into sentiment over the features of a Markov decision process. We then infer the teacher's reward function by regressing the sentiment on the features, an analogue of inverse reinforcement learning. To evaluate our approach, we first collect a corpus of teaching behavior in a cooperative task where both teacher and learner are human. We implement three artificial learners: sentiment-based "literal" and "pragmatic" models, and an inference network trained end-to-end to predict rewards. We then re-run our initial experiment, pairing human teachers with these artificial learners. All three models successfully learn from interactive human feedback. The inference network approaches the performance of the "literal" sentiment model, while the "pragmatic" model nears human performance. Our work provides insight into the information structure of naturalistic linguistic feedback as well as methods to leverage it for reinforcement learning.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Training Language Models with Language Feedback (2022, background): “…2017; Mahmoudieh et al., 2022; Ouyang et al., 2022, inter alia ), drive exploration (Tam et al., 2022), infer reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and train the model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017),…”
Interactive Learning from Activity Description (2021, background): “Common approaches include: reduction to reinforcement learning (Goldwasser & Roth, 2014; MacGlashan et al., 2015; Ling & Fidler, 2017; Goyal et al., 2019; Fu et al., 2019; Sumers et al., 2020), learning to ground language to actions (Chen & Mooney, 2011; Misra et al.”
Training Language Models with Language Feedback at Scale (2023, background): “…as specifying tasks (“instruction following” , inter alia ) driving exploration (Tam et al., 2022), inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and training a model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017),…”
The rise and potential of large language model based agents: a survey (2025, unknown): “For example, some studies [113, 198–200] employed reinforcement learning to perceive implied meanings and models feedback to derive rewards.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, unknown): “A body of work on using language feedback for reward inference and shaping might lessen this challenge [Fu et al., 2019, Goyal et al., 2019, Sumers et al., 2021, Zhou and Small, 2021, Lin et al., 2022, Yu et al., 2023], but thus far, these techniques have not been applied to LLMs.”