Combining manual feedback with subsequent MDP reward signals for reinforcement learning
By W. B. Knox, Peter Stone. 2010
242 citations recorded by the source corpus.
As learning agents move from research labs to the real world, it is increasingly important that human users, including those without programming skills, be able to teach agents desired behaviors. Recently, the tamer framework was introduced for designing agents that can be interactively shaped by human trainers who give only positive and negative feedback signals. Past work on tamer showed that shaping can greatly reduce the sample complexity required to learn a good policy, can enable lay users to teach agents the behaviors they desire, and can allow agents to learn within a Markov Decision Process (MDP) in the absence of a coded reward function. However, tamer does not allow this human training to be combined with autonomous learning based on such a coded reward function. This paper leverages the fast learning exhibited within the tamer framework to hasten a reinforcement learning (RL) algorithm's climb up the learning curve, effectively demonstrating that human reinforcement and MDP reward can be used in conjunction with one another by an autonomous agent. We tested eight plausible tamer+rl methods for combining a previously learned human reinforcement function, H, with MDP reward in a reinforcement learning algorithm. This paper identifies which of these methods are most effective and analyzes their strengths and weaknesses. Results from these tamer+rl algorithms indicate better final performance and better cumulative performance than either a tamer agent or an RL agent alone.
Subfields: Artificial Intelligence, Alignment & RLHF, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Stone
- Sergey Levine
- Pieter Abbeel
- Matthew E. Taylor
- Yoshua Bengio
- Chelsea Finn
- W. B. Knox
- Gerhard Neumann
In Alignment & RLHF
- Kristen Grauman
- Santhosh K. Ramakrishnan
- Julie Arnold Shah
- Stefanos Nikolaidis
- Przemyslaw A. Lasota
- Ramya Ramakrishnan
- Peter Stone
- W. B. Knox
In Reinforcement Learning
- Peter Stone
- Sergey Levine
- Matthew E. Taylor
- Pieter Abbeel
- Gerhard Neumann
- W. B. Knox
- Jan Peters
- Christian Wirth
Citations in the corpus
- Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “The TAMER+RL framework extends TAMER by learning from both human evaluative feedback and environment reward signal [16, 17].”
- Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “A natural extension is to combine human reward signal with environment reward signal, an approach named TAMER+RL [ Knox and Stone, 2010; Knox and Stone, 2012 ] .”
- Interactive Learning from Policy-Dependent Human Feedback (2017, background): “Some research has also examined combining human feed-back with more traditional environmental rewards (Knox & Stone, 2010; Tenorio-Gonzalez et al., 2010; Clouse & Utgoff, 1992; Maclin et al., 2005).”
- Multi-agent reinforcement learning as a rehearsal for decentralized planning (2016, neutral): “This work also differs from the large body of work on reward shaping where an agent explicitly engineers the reward function [17,18] to improve learning speed, or a human feedback is used to construct shaping rewards [13].”
- Reinforcement learning from simultaneous human and MDP reward (2012, methodology): “6 The models we create—ˆH 1 andˆH 2 —from the original training trajectories perform a bit better than those from previous experiments [8], which points to small implementation differences.”
- FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, background): “This work was extended to enable human feedback to augment an RL agent that learned using an MDP reward signal in [22, 23].”
- Reinforcement Learning with Limited Reinforcement : Using Bayes Risk for Active Learning in POMDPs Finale (2012, background): “Much of the work in inverse reinforcement learn ing assumes that states are fully-observable [18, 47, 48, 49, 50].”
- Deep Reinforcement Learning from Policy-Dependent Human Feedback (2019, unknown): “MDPs naturally give rise to two important functions.”
- Reinforcement Learning Via Practice and Critique Advice (2010, unknown): “This approach has shown some promise in user studies, but does not integrate autonomous practice (until recently; see (Knox and Stone 2010)) and is hence limited by the abilities of the user.”
- A comprehensive survey on safe reinforcement learning (2015, background): “Other works combine the reward function of the MDP and that provided by the teacher (Knox and Stone, 2010; Knox et al., 2011).”
- Imitation Learning with Demonstrations and Shaping Rewards (2014, background): “Other prior work allows a teacher to train an agent online by supplying numeric reward signals about an agent’s actions (Knox and Stone 2009; 2010; 2012; Thomaz and Breazeal 2008), sometimes in combination with the true environmental reward.”
- Dynamic Reward Shaping: Training a Robot by Voice (2010, background): “Some authors have provide traces from a user that can be used to guide the reinforcement learning process [1,8].”
- A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “Advice-based algorithms (Torrey and Taylor, 2013; Knox and Stone, 2010, 2012; Griffith et al., 2013) differ from PbRL in that the (possibly preference-based) advice is given in addition to a numeric reward signal.”
- Sidekick Policy Learning for Active Visual Exploration (2018, methodology): “The TAMER framework [33,34,35] utilizes expert human rewards about the end-task.”
- Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models (2020, background): “TAMER [25] and other interactive RL approaches assume a continuous user feedback mechanism that provides the RL reward, which is a significant burden on the user.”
- Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback (2017, background): “The Interactive Shaping Problem (ISP) defines the problem of optimizing the incorporation of human feedback into a learning agent in a sequential decision making problem (Knox & Stone, 2010).”