Leveraging Human Guidance for Deep Reinforcement Learning Tasks

By Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, Peter Stone. 2019

98 citations recorded by the source corpus.

Reinforcement learning agents can learn to solve sequential decision tasks by interacting with the environment. Human knowledge of how to solve these tasks can be incorporated using imitation learning, where the agent learns to imitate human demonstrated decisions. However, human guidance is not limited to the demonstrations. Other types of guidance could be more suitable for certain tasks and require less human effort. This survey provides a high-level overview of five recent learning frameworks that primarily rely on human guidance other than conventional, step-by-step action demonstrations. We review the motivation, assumption, and implementation of each framework. We then discuss possible future research directions.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Human-Computer Interaction

Citations in the corpus

Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “One paradigm is Reinforcement Learning (RL) with human feedback [40], where a human provides rewards during agent training [8, 11, 24, 25, 34, 36], but this suffers from the same limitations as IRL due to the need for extensive agent interaction.”
A Survey of Human-in-the-loop for Machine Learning (2021, background): “If the developer can encourage the machine to engage with learning human wisdom and knowledge, it would help deal with sparse data, especially in medical fields like clinical diagnosis and lack of training data [20, 21, 22, 23].”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, background): “ORK The role of feedback provided by a human to an agent in RL settings has been studied by multiple researchers. A survey of recent research in using human guidance for deep RL tasks is presented in [50]. We summarize related work in some of these techniques that are most relevant to this paper in ”
Human Gaze Assisted Artificial Intelligence: A Review (2020, methodology): “It was proposed that human gaze can be used as an auxiliary guiding signal in the imitation learning paradigm [Zhang et al. , 2019a].”
Unity: A General Platform for Intelligent Agents (2018, background): “A number of training frameworks have been studied in the literature (Zhang et al., 2019) such as learning to imitate expert trajectories (Ho and Ermon, 2016), humans providing evaluative feedback to the agent (Knox and Stone, 2008), or humans manipulating the agent’s observed states and actions…”
Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset (2019, background): “A major issue of RL is its sample inefﬁciency and human demonstration has been shown to speed up learning (Silver et al. 2016; Hester et al. 2018; de la Cruz, Du, and Taylor 2018; Zhang et al. 2019).”
Efficiently Guiding Imitation Learning Algorithms with Human Gaze (2020, methodology): “Gaze has been shown to improve performance of imitation learning algorithms (Zhang et al. 2019),”