Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation

By Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, Subbarao Kambhampati. 2020

56 citations recorded by the source corpus.

Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative"good"or"bad"feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.

Subfields: Reinforcement Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Human-Computer Interaction

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “[116] improved sample efficiency by incorporating domain knowledge through human feedback and visual explanations.”
Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems (2021, unknown): “Object-oriented symbolic interfaces like these have also been used in other previous works to allow humans to provide informative and generalizable object-focused ad-vices in an effortless way (Thomaz, Breazeal et al. 2006; Krening et al. 2016; Guan et al. 2021).”
Tell me why! Explanations support learning relational and causal structure (2021, background): “In RL, however, applications of feature explanations have been more limited, although Guan et al. (2021) used human annotations of relevant visual features (and binary feedback) to generate augmentations that varied the task-irrelevant features, and showed beneﬁts over other feature-based…”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “There have been a few explorations within reinforcement learning (RL; Guan et al., 2021; Tulli et al., 2020), but these have not focused on relational or causal reasoning.”