The Primacy Bias in Deep Reinforcement Learning
By Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, Aaron C. Courville. International Conference on Machine Learning, 2022
307 citations recorded by the source corpus.
This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias. Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/69b80ce5ab6c2263b145b1eaa23664145938f351
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Anima Anandkumar
- Sergey Levine
- Yuke Zhu
- De-An Huang
- Linxi (Jim) Fan
- Ajay Mandlekar
- Guanzhi Wang
- Tim Rocktaschel
Citations in the corpus
- Loss of plasticity in deep continual learning (2024, unknown): “The effectiveness of continual backpropagation shows that the problem of plasticity loss is not inherent in artificial neural networks.”
- MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “We also follow the recommended practice in [83] and reset the policy head at the beginning of stage 2 to encourage exploration and reduce overfitting.”
- Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “We speculate that this is due to bad representations learned early on by the student agent (Nikishin et al., 2022; Cetin et al., 2022), which can be avoided by using distillation.”