Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces

By Garrett Warnell, Nicholas R. Waytowich, Vernon J. Lawhern, Peter Stone. 2017

303 citations recorded by the source corpus.

While recent advances in deep reinforcement learning have allowed autonomous learning agents to succeed at a variety of complex tasks, existing algorithms generally require a lot oftraining data. One way to increase the speed at which agent sare able to learn to perform tasks is by leveraging the input of human trainers. Although such input can take many forms, real-time, scalar-valued feedback is especially useful in situations where it proves difficult or impossible for humans to provide expert demonstrations. Previous approaches have shown the usefulness of human input provided in this fashion (e.g., the TAMER framework), but they have thus far not considered high-dimensional state spaces or employed the use of deep learning. In this paper, we do both: we propose DeepTAMER, an extension of the TAMER framework that leverages the representational power of deep neural networks inorder to learn complex tasks in just a short amount of time with a human trainer. We demonstrate Deep TAMER’s success by using it and just 15 minutes of human-provided feedback to train an agent that performs better than humans on the Atari game of Bowling - a task that has proven difficult for even state-of-the-art reinforcement learning methods.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Besides learning to use external tools, RLHF has also proven useful for a wide range of language generation tasks, from summarization (Ziegler et al., 2019; Wu et al., 2021; Stiennon et al., 2020) to training more helpful, harmless, and accurate assistants (Glaese et al., 2022; Cohen et al., 2022; O”
Scalable agent alignment via reward modeling: a research direction (2018, background): “TAMER (Knox & Stone, 2009; Knox, 2012; Warnell et al., 2017) is a collection of algorithms that learn a policy from human value feedback, i.e. take actions that maximize expected feedback in the next step (possibly with short temporal smoothing).”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “, 2008), policy feedback (Knox and Stone, 2009; Warnell et al., 2017), or trajectory preferences (Wilson et al.”
Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “Besides IRL-style approaches, other apprenticeship learning methods involve training a policy (Knox & Stone, 2009; Warnell et al., 2017) or a reward function (Wilson et al., 2012; Christiano et al., 2017) directly from human feedback.”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “Similar to Deep-TAMER [45], we also applied each received feedback and explanation to frames that are displayed between 2 and 0.2 seconds before the feedback occurred – we assume that within this 1.8-second window, the salient regions/objects should be the same.”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “The TAMER (Training an Agent Manually via Evaluative Reinforcement) framework adopts the reward function interpretation [ Knox and Stone, 2009 ] and has been recently extended to deep TAMER [ Warnell et al. , 2018 ] .”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “Many advanced agent training methods, including DAGGER (Ross et al., 2011), TAMER (Warnell et al., 2017) and learning from human preferences (Wilson et al., 2012; Christiano et al., 2017), assume that interaction between the learner and the teacher is possible.”
Active Reinforcement Learning with Monte-Carlo Tree Search (2018, background): “In Reward Learning, a reward function is learned online from human evaluations of the agent’s behaviour (Warnell et al., 2017; Christiano et al., 2017; Saunders et al., 2017).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “In order to learn complex behavior, human trainer feedbacks has the potential to be more performant than a reward function defined a priori (MacGlashan et al., 2017; Warnell et al., 2017).”
Measuring and avoiding side effects using relative reachability (2018, background): “…effects through human oversight, such as inverse reinforcement learning (Ng and Russell, 2000; Ziebart et al., 2008; Hadfield-Menell et al., 2016), demonstrations (Abbeel and Ng, 2004; Hester et al., 2018), or human feedback (Christiano et al., 2017; Saunders et al., 2017; Warnell et al., 2018).”
Preference Ranking Optimization for Human Alignment (2023, background): “Many explorations have been done on this path (Chris-tiano et al., 2017; MacGlashan et al., 2017; Warnell et al., 2018; Ziegler et al., 2019; Stiennon et al., 2020b; Nakano et al., 2021; Lee et al., 2021; Lei et al., 2022; Snell et al., 2022; Bai et al., 2022a; Ouyang et al., 2022a).”
Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks (2020, background): “Knowledge from a human trainer can be transferred through real-time feedback to improve learning performance (Thomaz and Breazeal 2006; Knox and Stone 2009; Warnell et al. 2018).”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “Others involve training a policy [15, 26] or a reward function [27, 8] directly from human feedback.”
Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments (2019, background): “Deep reinforcement learning with human evaluative feedback has also been shown to quickly train policies across a variety of domains [18, 34] and can be a particularly useful approach when the human is unable to provide a demonstration of desired behavior but can articulate when desired behavior is ”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, methodology): “There has been a large amount of empirical work that demonstrates the success of MLE and pessimistic MLE in RLHF for game playing (Knox and Stone, 2008; MacGlashan et al., 2017; Christiano et al., 2017; Warnell et al., 2018), robotics (Brown et al.”
Learning to Interactively Learn and Assist (2019, background): “Examples of such interactions include allowing user interventions or agent requests, for demonstrations (Kelly et al. 2018), rewards (Warnell et al. 2018; Arumugam et al. 2019), or preferences (Christiano et al. 2017).”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, methodology): “More recently, the authors of [45] proposed Deep-TAMER, an architecture that extends the TAMER framework to environments with high-dimensional state spaces.”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, methodology): “sion using unbounded real-valued feedback (Knox & Stone, 2009; Warnell et al., 2018), but this has been challenging to scale because it is difficult for humans to reliably provide a particular utility value for certain behaviors of the RL agent.”
DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback (2018, methodology): “This method is called human-inthe-loop RL and its effectiveness has been reported in prior publications [6]–[15].”