FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

By Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda G. Bushnell, Radha Poovendran. 2020

28 citations recorded by the source corpus.

Reinforcement learning has been successful in training autonomous agents to accomplish goals in complex environments. Although this has been adapted to multiple settings, including robotics and computer games, human players often find it easier to obtain higher rewards in some environments than reinforcement learning algorithms. This is especially true of high-dimensional state spaces where the reward obtained by the agent is sparse or extremely delayed. In this paper, we seek to effectively integrate feedback signals supplied by a human operator with deep reinforcement learning algorithms in high-dimensional state spaces. We call this FRESH (Feedback-based REward SHaping). During training, a human operator is presented with trajectories from a replay buffer and then provides feedback on states and actions in the trajectory. In or- der to generalize feedback signals provided by the human operator to previously unseen states and actions at test-time, we use a feed-back neural network. We use an ensemble of neural networks with a shared network architecture to represent model uncertainty and the confidence of the neural network in its output. The output of the feedback neural network is converted to a shaping reward that is augmented to the reward provided by the environment. We evaluate our approach on the Bowling and Skiing Atari games in the arcade learning environment. Although human experts have achieved high scores in these environments, state-of-the-art deep learning algorithms perform poorly. We observe that FRESH achieves much higher scores than state-of-the-art deep learning algorithms in both environments. FRESH also achieves a 21.4% higher score than a human expert in Bowling and does as well as an expert in Skiing.

Subfields: Artificial Intelligence, Automotive Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lin Guan
2. Ruohan Zhang
3. Mudit Verma
4. Subbarao Kambhampati
5. Sihang Guo
6. Sarath Sreedharan

Citations in the corpus

• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “, where human trainers are required to be more actively involved during training [2, 47, 8].”