Agent-Agnostic Human-in-the-Loop Reinforcement Learning

By David Abel, John Salvatier, Andreas Stuhlmüller, Owain Evans. 2017

66 citations recorded by the source corpus.

Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. Prior work has developed teaching protocols that enable agents to learn efficiently in complex environments; many of these methods tailor the teacher's guidance to agents with a particular representation or underlying learning scheme, offering effective but specialized teaching procedures. In this work, we explore protocol programs, an agent-agnostic schema for Human-in-the-Loop Reinforcement Learning. Our goal is to incorporate the beneficial properties of a human teacher into Reinforcement Learning without making strong assumptions about the inner workings of the agent. We show how to represent existing approaches such as action pruning, reward shaping, and training in simulation as special cases of our schema and conduct preliminary experiments on simple domains.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tom Everitt
2. Jan Leike
3. Shane Legg
4. David Krueger
5. Miljan Martic
6. Sergey Levine
7. Peter Stone
8. Ruohan Zhang

In Control and Systems Engineering

1. Jan Leike
2. Ruohan Zhang
3. Faraz Torabi
4. Peter Stone
5. Tom Everitt
6. Shane Legg
7. Lin Guan
8. Dana H. Ballard

In Reinforcement Learning

1. Tom Everitt
2. Jan Leike
3. Shane Legg
4. David Krueger
5. Miljan Martic
6. Ruohan Zhang
7. Owain Evans
8. Peter Stone

Citations in the corpus

• Scalable agent alignment via reward modeling: a research direction (2018, background): “The user could proactively provide off-policy feedback in anticipation of potential pitfalls (Abel et al., 2017).”
• Reinforcement Learning in Healthcare: A Survey (2019, background): “Another way of considering prior knowledge can make use of human cognitive abilities or domain expertise to guide, shape, evaluate, or validate the agent’s learning process, making the traditional RL into a human-in-the-loop interactive RL problem [1].”
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, background): “Andreas et al. take a similar hierarchical approach but only require humans to provide policy sketches that are high-level symbolic subtask labels [2017].”
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, methodology): “HIRL is agnostic as to the inner workings of the RL algorithm (building on our earlier work [1]).”
• Unity: A General Platform for Intelligent Agents (2018, background): “or humans manipulating the agent’s observed states and actions (Abel et al., 2016).”
• How RL Agents Behave When Their Actions Are Modified (2021, background): “Dalal et al. [5] and Abel et al. [1] both propose approaches that construct a wrapper around an inner reinforcement learning agent.”
• Bridging the Gap: Converting Human Advice into Imagined Examples (2019, methodology): “Use of simulations for training RL agents was used in work such as Abel et al. (2016).”