Reinforcement learning from simultaneous human and MDP reward

By W. B. Knox, Peter Stone. 2012

202 citations recorded by the source corpus.

As computational agents are increasingly used beyond research labs, their success will depend on their ability to learn new skills and adapt to their dynamic, complex environments. If human users---without programming skills---can transfer their task knowledge to agents, learning can accelerate dramatically, reducing costly trials. The tamer framework guides the design of agents whose behavior can be shaped through signals of approval and disapproval, a natural form of human feedback. More recently, tamer+rl was introduced to enable human feedback to augment a traditional reinforcement learning (RL) agent that learns from a Markov decision process's (MDP) reward signal. We address limitations of prior work on tamer and tamer+rl, contributing in two critical directions. First, the four successful techniques for combining human reward with RL from prior tamer+rl work are tested on a second task, and these techniques' sensitivities to parameter changes are analyzed. Together, these examinations yield more general and prescriptive conclusions to guide others who wish to incorporate human knowledge into an RL algorithm. Second, tamer+rl has thus far been limited to a sequential setting, in which training occurs before learning from MDP reward. In this paper, we introduce a novel algorithm that shares the same spirit as tamer+rl but learns simultaneously from both reward sources, enabling the human feedback to come at any time during the reinforcement learning process. We call this algorithm simultaneous tamer+rl. To enable simultaneous learning, we introduce a new technique that appropriately determines the magnitude of the human model's influence on the RL algorithm throughout time and state-action space.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “The TAMER+RL framework extends TAMER by learning from both human evaluative feedback and environment reward signal [16, 17].”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, background): “A natural extension is to combine human reward signal with environment reward signal, an approach named TAMER+RL [ Knox and Stone, 2010; Knox and Stone, 2012 ] .”
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces (2017, methodology): “This observation was recently made and exploited in a series of papers outlining the TAMER framework (Knox and Stone 2009; 2012; Knox, Stone, and Breazeal 2013; Knox and Stone 2015) (see Figure 1), in which it was shown that agents learning directly from non-expert human trainers can improve over…”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, background): “This work was extended to enable human feedback to augment an RL agent that learned using an MDP reward signal in [22, 23].”
Sidekick Policy Learning for Active Visual Exploration (2018, methodology): “The TAMER framework [33,34,35] utilizes expert human rewards about the end-task.”
Human-robot cross-training: Computational formulation, modeling and evaluation of a human team training strategy (2013, methodology): “Under this formulation, the role of the robot is represented by the optimal policy π ∗ , whereas the robot’s knowledge of the role of the human co-worker is represented by the transition probabilities T .”
Learning from Limited Demonstrations (2013, methodology): “The TAMER system [24] uses rewards provided by the expert (and possibly blended with MDP rewards), instead of assuming that an action choice is provided.”
Policy Shaping: Integrating Human Feedback with Reinforcement Learning (2013, methodology): “The issue of credit assignment that naturally arises with learning from real human feedback is left for future work (see [13] for an implementation of credit assignment in a different framework for learning from human feedback).”
Teaching Machines to Describe Images via Natural Language Feedback (2017, background): “Several works explored the idea of incorporating humans in the learning process, in order to help the reinforcement learning agent to learn faster [35, 12, 11, 6, 5].”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, methodology): “Published research on Interactive Reinforcement Learning (Thomaz and Breazeal, 2006; Knox and Stone, 2009; Suay and Chernova, 2011; Knox and Stone, 2012; Griffith et al., 2013; Subramanian et al., 2016; Loftin et al., 2016; MacGlashan et al., 2017) has shown that RL agents can successfully speed up ”
Exploration from Demonstration for Interactive Reinforcement Learning (2016, unknown): “Work by Knox and Stone [22, 23] has shown that a general improvement to learning from human feed-back is possible if it is used to directly modify the action selection mechanism of the Reinforcement Learning algo-rithm.”
Shared Autonomy via Deep Reinforcement Learning (2018, unknown): “In contrast, human-in-the-loop reinforcement learning frameworks leverage human feedback to train autonomous agents that operate independently of the user at test time [32, 14, 15, 19].”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “Knox and Stone (2012) and Thomaz and Breazeal (2006) analyzed this problem in an advice setting and observed that feedback relating to immediate rewards are difficult to use and feedback concerning the expected, long-term return should be preferred.”
Improved human–robot team performance through cross-training, an approach inspired by human team training practices (2015, methodology): “Note that our goal is not to examine the best way to update the rewards, as this has proven to be task-dependent (Knox and Stone, 2012).”