Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation
By Tiantian Zhang, Xueqian Wang, Bin Liang, Bo Yuan. 2021
27 citations recorded by the source corpus.
The powerful learning ability of deep neural networks enables reinforcement learning (RL) agents to learn competent control policies directly from continuous environments. In theory, to achieve stable performance, neural networks assume identically and independently distributed (i.i.d.) inputs, which unfortunately does not hold in the general RL paradigm where the training data are temporally correlated and nonstationary. This issue may lead to the phenomenon of “catastrophic interference” and the collapse in performance. In this article, we present interference-aware deep Q-learning (IQ) to mitigate catastrophic interference in single-task deep RL. Specifically, we resort to online clustering to achieve on-the-fly context division, together with a multihead network and a knowledge distillation regularization term for preserving the policy of learned contexts. Built upon deep Q networks (DQNs), IQ consistently boosts the stability and performance when compared to existing methods, verified with extensive experiments on classic control and Atari tasks. The code is publicly available at https://github.com/ Sweety-dm/Interference-aware-Deep-Q-learning.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- Sergey Levine
- Joelle Pineau
- Eric Eaton
- Chelsea Finn
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2015) from past tasks when learning a new task to prevent catastrophic forgetting as in (Rusu et al., 2015; Li and Hoiem, 2016; Riemer et al., 2016; Espeholt et al., 2018; Schwarz et al., 2018; Berseth et al., 2018; Kaplanis et al., 2019; Traoré et al., 2019; Tirumala et al., 2019; Igl et al., 202”