DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

By Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, Shin-ichi Maeda. 2018

92 citations recorded by the source corpus.

Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ruohan Zhang
2. Lin Guan
3. Faraz Torabi
4. Peter Stone
5. Dana H. Ballard
6. Hongsheng Li
7. Subbarao Kambhampati
8. Sihang Guo

In Cognitive Neuroscience

1. Ruohan Zhang
2. Lin Guan
3. Peter Stone
4. Rui Zhao
5. Dana H. Ballard
6. Feng Zhu
7. Faraz Torabi
8. Hongsheng Li

In Electrical and Electronic Engineering

1. Radha Poovendran
2. Peter Stone
3. Dana H. Ballard
4. Ruohan Zhang
5. Hongsheng Li

Citations in the corpus

• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “To answer the first question, we compare EXPAND to DQN-Feedback and an HIRL algorithm DQN-TAMER which combines TAMER with deep RL [2].”
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “For example, human voice [ Tenorio-Gonzalez et al. , 2010 ] and facial expression [ Arakawa et al. , 2018 ] can be used as additional sources of evaluative feed-back.”
• Human Preference Score: Better Aligning Text-to-image Models with Human Preference (2023, background): “[1] incorporate human feedback into RL training, which is proven to accelerate the model convergence.”
• FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, background): “Another extension of TAMER, DQN-TAMER [1], modeled additional characteristics of human observers like inferring human reward from facial expressions.”