Understanding Regularisation Methods for Continual Learning
By Frederik Benzing. 2020
9 citations recorded by the source corpus.
The problem of Catastrophic Forgetting has received a lot of attention in the past years. An important class of proposed solutions are so-called regularisation approaches, which protect weights from large changes according to their importances. Various ways to measure this importance have been put forward, all stemming from different theoretical or intuitive motivations. We present mathematical and empirical evidence that two of these methods -- Synaptic Intelligence and Memory Aware Synapses -- approximate a rescaled version of the Fisher Information, a theoretically justified importance measure also used in the literature. As part of our methods, we show that the importance approximation of Synaptic Intelligence is biased and that, in fact, this bias explains its performance best. Altogether, our results offer a theoretical account for the effectiveness of different regularisation approaches and uncover similarities between the methods proposed so far.
Subfields: Artificial Intelligence, Computational Mechanics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mehdi Abbana Bennani
- Masashi Sugiyama
- Amos J. Storkey
- K. Müller
- Balaji Lakshminarayanan
- Qiang Yang
- Bernhard Schölkopf
- Shotaro Akaho
In Computational Mechanics
Citations in the corpus
- Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, methodology): “asks in advance. Knoblauch et al. (2020) show that optimal Continual Learning algorithms generally solve an NP-HARD problem and will require perfect memory not to suffer from catastrophic forgetting. Benzing (2020) presents mathematical and empirical evidence that the two methods – Synaptic Intellig”