Combining manual feedback with subsequent MDP reward signals for reinforcement learning

By W. B. Knox, Peter Stone. 2010

242 citations recorded by the source corpus.

As learning agents move from research labs to the real world, it is increasingly important that human users, including those without programming skills, be able to teach agents desired behaviors. Recently, the tamer framework was introduced for designing agents that can be interactively shaped by human trainers who give only positive and negative feedback signals. Past work on tamer showed that shaping can greatly reduce the sample complexity required to learn a good policy, can enable lay users to teach agents the behaviors they desire, and can allow agents to learn within a Markov Decision Process (MDP) in the absence of a coded reward function. However, tamer does not allow this human training to be combined with autonomous learning based on such a coded reward function. This paper leverages the fast learning exhibited within the tamer framework to hasten a reinforcement learning (RL) algorithm's climb up the learning curve, effectively demonstrating that human reinforcement and MDP reward can be used in conjunction with one another by an autonomous agent. We tested eight plausible tamer+rl methods for combining a previously learned human reinforcement function, H, with MDP reward in a reinforcement learning algorithm. This paper identifies which of these methods are most effective and analyzes their strengths and weaknesses. Results from these tamer+rl algorithms indicate better final performance and better cumulative performance than either a tamer agent or an RL agent alone.

Subfields: Artificial Intelligence, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Peter Stone
  2. Sergey Levine
  3. Pieter Abbeel
  4. Matthew E. Taylor
  5. Yoshua Bengio
  6. Chelsea Finn
  7. W. B. Knox
  8. Gerhard Neumann

In Alignment & RLHF

  1. Kristen Grauman
  2. Santhosh K. Ramakrishnan
  3. Julie Arnold Shah
  4. Stefanos Nikolaidis
  5. Przemyslaw A. Lasota
  6. Ramya Ramakrishnan
  7. Peter Stone
  8. W. B. Knox

In Reinforcement Learning

  1. Peter Stone
  2. Sergey Levine
  3. Matthew E. Taylor
  4. Pieter Abbeel
  5. Gerhard Neumann
  6. W. B. Knox
  7. Jan Peters
  8. Christian Wirth

Citations in the corpus