Interactively shaping agents via human reinforcement: the TAMER framework

By W. B. Knox, Peter Stone. 2009

590 citations recorded by the source corpus.

As computational learning agents move into domains that incur real costs (e.g., autonomous driving or financial investment), it will be necessary to learn good policies without numerous high-cost learning trials. One promising approach to reducing sample complexity of learning a task is knowledge transfer from humans to agents. Ideally, methods of transfer should be accessible to anyone with task knowledge, regardless of that person's expertise in programming and AI. This paper focuses on allowing a human trainer to interactively shape an agent's policy via reinforcement signals. Specifically, the paper introduces "Training an Agent Manually via Evaluative Reinforcement," or TAMER, a framework that enables such shaping. Differing from previous approaches to interactive shaping, a TAMER agent models the human's reinforcement and exploits its model by choosing actions expected to be most highly reinforced. Results from two domains demonstrate that lay users can train TAMER agents without defining an environmental reward function (as in an MDP) and indicate that human training within the TAMER framework can reduce sample complexity over autonomous learning algorithms.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “MacGlashan et al. (2017), Pilarski et al. (2011), Knox and Stone (2009), and Knox (2012) perform experiments involving reinforcement learning from actual human feedback, although their algorithmic approach is less similar.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…& Ng, 2004; Argall et al., 2009); providing feedback in the form of scores (El Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012; Akrour et al., 2012, 2014; Wirth et…”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “There is a large body of work on reinforcement learning from human ratings or rankings (Wirth et al., 2017): Knox and Stone (2009), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wirth et al. (2016),…”
Supervising strong learners by amplifying weak experts (2018, background): “…box (RL from a human training signal) is beginning to be explored, and includes inverse reinforcement learning (Ng and Russell, 2000; Abbeel and Ng, 2004; Finn et al., 2016) and RL from human feedback (Knox and Stone, 2009; Pilarski et al., 2011; MacGlashan et al., 2017; Christiano et al., 2017).”
Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “Besides IRL-style approaches, other apprenticeship learning methods involve training a policy (Knox & Stone, 2009; Warnell et al., 2017) or a reward function (Wilson et al., 2012; Christiano et al., 2017) directly from human feedback.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, background): “our work to previous work on RL from human rewards (Knox and Stone, 2009; Christiano et al., 2017) is that our RL scenario is not interactive, but rewards are collected in an offline log.”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “One popular form of human guidance in HIRL is the binary evaluative feedback [15], in which humans provide a "good" or "bad" judgment for a queried state-action pair.”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “The TAMER (Training an Agent Manually via Evaluative Reinforcement) framework adopts the reward function interpretation [ Knox and Stone, 2009 ] and has been recently extended to deep TAMER [ Warnell et al. , 2018 ] .”
Interactive Learning from Policy-Dependent Human Feedback (2017, methodology): “In TAMER (Knox & Stone, 2009a), a trainer provides interactive numeric feedback that is interpreted as an exemplar of the reward function for the demonstrated state–action pair as the learner takes actions.”
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces (2017, methodology): “For TAMER, we implemented Algorithm 2 of (Knox and Stone 2009) using the same credit assignment scheme, with the only difference being that, simi-”
Active Reward Learning (2014, methodology): “Derived from this terminology the TAMER framework [19] uses the term interactive shaping.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, unknown): “Human preference learning trains models on judgements of which behaviors people prefer, rather than on human demonstrations directly (Knox & Stone, 2009; Akrour et al., 2011).”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “As such, human-in-the-loop (HiL) RL (Knox & Stone, 2009; Christiano et al., 2017; MacGlashan et al., 2017) has not yet been widely adopted.”
A comprehensive survey on safe reinforcement learning (2015, background): “by pressing a button or verbal feedback of “good” and “bad”) (Thomaz and Breazeal, 2006; Knox and Stone, 2009).”
Agent-Agnostic Human-in-the-Loop Reinforcement Learning (2017, methodology): “This setup has been explored in work on TAMER [23].”
Integrating reinforcement learning with human demonstrations of varying ability (2011, background): “Their TAMER [8] system learns to predict and maximize a reward that is interactively provided by a human.”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “Others involve training a policy [15, 26] or a reward function [27, 8] directly from human feedback.”
Learning From Stories: Using Crowdsourced Narratives to Train Virtual Agents (2021, background): “These systems use human reward signals to shape agent behavior (Knox and Stone 2009; Judah et al. 2010; Loftin et al. 2014).”
Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning (2015, methodology): “M−0 is designed to be most similar to the TAMER framework [17] and ignores episodes without feedback, making no changes to its value estimates in that case.”