Continual Backprop: Stochastic Gradient Descent with Persistent Randomness
By Shibhansh Dohare, A. Mahmood, Richard S. Sutton. 2021
104 citations recorded by the source corpus.
The Backprop algorithm for learning in neural networks utilizes two mechanisms: first, stochastic gradient descent and second, initialization with small random weights, where the latter is essential to the effectiveness of the former. We show that in continual learning setups, Backprop performs well initially, but over time its performance degrades. Stochastic gradient descent alone is insufficient to learn continually; the initial randomness enables only initial learning but not continual learning. To the best of our knowledge, ours is the first result showing this degradation in Backprop's ability to learn. To address this degradation in Backprop's plasticity, we propose an algorithm that continually injects random features alongside gradient descent using a new generate-and-test process. We call this the \textit{Continual Backprop} algorithm. We show that, unlike Backprop, Continual Backprop is able to continually adapt in both supervised and reinforcement learning (RL) problems. Continual Backprop has the same computational complexity as Backprop and can be seen as a natural extension of Backprop for continual learning.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- A. Rupam Mahmood
- Mohamed Elsayed
- Sergey Levine
- Joelle Pineau
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Finally, Ajemian, D’Ausilio, Moorman, and Bizzi (2013) demonstrate the role that noisy computations in the human brain may have in avoiding catastrophic forgetting, and Dohare, Mahmood, and Sutton (2021) also demonstrate that noisy computations can improve plasticity.”
- Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (2024, methodology): “Recent studies reveal that SGD or Adam continues to lose plasticity with more tasks, primarily due to features becoming difficult to modify (Dohare et al. 2021, Lyle et al. 2023).”