Interactive Learning from Policy-Dependent Human Feedback

By James MacGlashan, Mark K. Ho, Robert Tyler Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, Michael L. Littman. 2017

346 citations recorded by the source corpus.

This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner's current policy. We present empirical results that show this assumption to be false— whether human trainers give a positive or negative feedback for a decision is influenced by the learner's current policy. Based on this insight, we introduce Convergent Actor-Critic by Humans (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.

Subfields: Artificial Intelligence, Control and Systems Engineering, Social Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Social Psychology

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Besides learning to use external tools, RLHF has also proven useful for a wide range of language generation tasks, from summarization (Ziegler et al., 2019; Wu et al., 2021; Stiennon et al., 2020) to training more helpful, harmless, and accurate assistants (Glaese et al., 2022; Cohen et al., 2022; O”
Deep Reinforcement Learning from Human Preferences (2017, background): “In MacGlashan et al. (2017) and Pilarski et al. (2011), learning only occurs during episodes where the human trainer provides feedback.”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “COACH (MacGlashan et al., 2017; Arumugam et al., 2018) is an algorithm that trains a policy from feedback in the form of an advantage function (Sutton & Barto, 2018).”
Reward learning from human preferences and demonstrations in Atari (2018, background): “ring in particular directions. However, instead of encouraging exploration, this feedback produces ‘reward pits’ that trap the agent into repetitive and fruitless behaviors. This effect is not novel; MacGlashan et al. (2017) have previously argued that humans are bad at shaping reward. However, our ”
Supervising strong learners by amplifying weak experts (2018, background): “…box (RL from a human training signal) is beginning to be explored, and includes inverse reinforcement learning (Ng and Russell, 2000; Abbeel and Ng, 2004; Finn et al., 2016) and RL from human feedback (Knox and Stone, 2009; Pilarski et al., 2011; MacGlashan et al., 2017; Christiano et al., 2017).”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “The actor-critic framework has been adapted to interactive RL from human feedback by Pilarski et al. (2011) and MacGlashan et al. (2017).”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “One paradigm is Reinforcement Learning (RL) with human feedback [40], where a human provides rewards during agent training [8, 11, 24, 25, 34, 36], but this suffers from the same limitations as IRL due to the need for extensive agent interaction.”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “The human feedback can then be used to replace the advantage function in calculating the policy gradient using the actor-critic algo-rithm [MacGlashan et al. , 2017].”
AI Safety Gridworlds (2017, background): “, 2017), and learning from human reward feedback (Akrour et al., 2012; Wilson et al., 2012; MacGlashan et al., 2017; Christiano et al., 2017), among others.”
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023, background): “These work can be categoried as supervised fine-tuing or reward modeling on well-constructed human annotations information, the latter is also known as a vital phase in RL from human feedback (Christiano et al., 2023; MacGlashan et al., 2017).”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, background): “…2006; Knox and Stone, 2009; Suay and Chernova, 2011; Knox and Stone, 2012; Griffith et al., 2013; Subramanian et al., 2016; Loftin et al., 2016; MacGlashan et al., 2017) has shown that RL agents can successfully speed up learning using human feedback, demonstrating the significant role can…”
Bridging the Gap: Converting Human Advice into Imagined Examples (2019, background): “MacGlashan et al. (2017) interpret human feedback in a policydependent manner by passing human feedback through the advantage function, which estimates the value of an action compared to the current policy.”
Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments (2019, background): “Deep reinforcement learning with human evaluative feedback has also been shown to quickly train policies across a variety of domains [18, 34] and can be a particularly useful approach when the human is unable to provide a demonstration of desired behavior but can articulate when desired behavior is ”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “…has been a large amount of empirical work that demonstrates the success of MLE and pessimistic MLE in RLHF for game playing (Knox and Stone, 2008; MacGlashan et al., 2017; Christiano et al., 2017a; Warnell et al., 2018), robotics (Brown et al., 2019; Shin et al., 2023) and language models…”
Learning Rewards from Linguistic Feedback (2020, methodology): “The RL literature has previously elicited such feedback (+1/-1) from human teachers (Thomaz and Breazeal 2008; Knox and Stone 2009; MacGlashan et al. 2017).”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, background): “as a scalar reward signal for a reinforcement learning agent, however empirical studies show that humans reward and punish based on how well the robot performs relative to their expectations [48].”