Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds

By Zhiyu Lin, Brent Harrison, Aaron Keech, Mark O. Riedl. 2017

39 citations recorded by the source corpus.

We describe a method to use discrete human feedback to enhance the performance of deep learning agents in virtual three-dimensional environments by extending deep-reinforcement learning to model the confidence and consistency of human feedback. This enables deep reinforcement learning algorithms to determine the most appropriate time to listen to the human feedback, exploit the current policy model, or explore the agent's environment. Managing the trade-off between these three strategies allows DRL agents to be robust to inconsistent or intermittent human feedback. Through experimentation using a synthetic oracle, we show that our technique improves the training speed and overall performance of deep reinforcement learning in navigating three-dimensional environments using Minecraft. We further show that our technique is robust to highly innacurate human feedback and can also operate when no human feedback is given.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Sergey Levine
3. Jason Aaron Edward Weston
4. Kyunghyun Cho
5. Douwe Kiela
6. Yoshua Bengio
7. Jianfeng Gao
8. Luke S. Zettlemoyer

In Automotive Engineering

1. Sergey Levine
2. Eric
3. Yeh
4. Armand Joulin
5. Anca Dragan
6. Pieter Abbeel
7. Ofir Nachum
8. Shixiang Shane Gu

In Reinforcement Learning

1. Anca Dragan
2. Sergey Levine
3. Pieter Abbeel
4. Shane Legg
5. Jan Leike
6. Chelsea Finn
7. Arthur Szlam
8. Geoffrey Irving

Citations in the corpus

• Reward learning from human preferences and demonstrations in Atari (2018, background): “Lin et al. (2017) apply deep RL from human feedback to 3D environments and improve the handling of low-quality or intermittent feedback.”
• Shared Autonomy via Deep Reinforcement Learning (2018, background): “In contrast, human-in-the-loop reinforcement learning frameworks leverage human feedback to train autonomous agents that operate independently of the user at test time [32, 14, 15, 19].”
• Bridging the Gap: Converting Human Advice into Imagined Examples (2019, methodology): “2. http://minecraft.net As learning basic percepts from visual input has proven to be a challenge for complex game environments (de la Cruz et al., 2017; Lin et al., 2017), we used observations of the block identities within a rectangular volume centered on the agent.”
• Why Build an Assistant in Minecraft? (2019, background): “The work of [55] considers the use of human feedback, but at the action level, not by via language.”