Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

By Xialei Liu, Marc Masana, Luis Herranz, Joost van de Weijer, Antonio M. López, Andrew D. Bagdanov. 2018

327 citations recorded by the source corpus.

In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to the state-of-the-art in lifelong learning without forgetting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Since the diagonal approximation of the FIM in EWC [49] might lose information about the old tasks, R-EWC [108] performs a factorized rotation of the parameter space to diagonalize the FIM.”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “Elastic weight consolidation ( EWC ; Kirkpatrick et al., 2017) and its successors (Chaudhry et al., 2018; Lee et al., 2017; Liu et al., 2018; Schwarz et al., 2018) use a Fisher information matrix computed on each task.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “…are now many existing methods for (deep) continual learning, including regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Liu et al., 2018; Ritter et al., 2018; Lesort et al., 2019; Park et al., 2019; Yin et al., 2020; Zhou et al., 2021; Heckel, 2022), memory-based…”
Long-Tailed Class Incremental Learning (2022, methodology): “R-EWC [22], Synaptic Intelligence (SI) [45], and Memory Aware Synapses (MAS) [3] adopt the same strategy but with different techniques to identify important weights.”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, background): “Naturally, existing works have focused on efficient representation of the matrix via diagonal approximation [18, 25] or Kronecker factorization [22, 38].”
Prototype-Guided Continual Adaptation for Class-Incremental Unsupervised Domain Adaptation (2022, methodology): “Step 3 [20,21,22,23,24,25,26,27,28,29] Postit Notes, Mug, Webcam, Desk Lamp, Telephone, Helmet, Mouse, Pen, Monitor, Mop”
Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “Although several methods effective at reducing forgetting when learning new tasks were proposed (Kirkpatrick et al., 2017; Lee et al., 2017; Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Masse et al., 2018; Liu et al., 2018; Rolnick et al.,”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[21] proposed to approximately diagonalize the Fisher information matrix by rotating the parameter space of the model, leaving the forward output unchanged.”
Rehearsal revealed: The limits and merits of revisiting samples in continual learning (2021, neutral): “In this section we compare rehearsal to pioneering work in this family, Elastic Weight Consolidation [18], inspiring many other works [42, 1, 23, 20].”
Rainbow Memory: Continual Learning with a Memory of Diverse Samples (2021, background): “There have been many proposals that can be roughly categorized into (1) rehearsal-based approaches [6, 4, 39], where episodic memory stores a few exemplars of old tasks, then the exemplars will be replayed in the future task, and (2) regularization-based approaches [28, 50, 33, 31, 29, 36], where no”
Few-Shot Lifelong Learning (2021, methodology): “Regularization-based methods try to reduce changes in the output logits/ important parameters of the network while training on new tasks in order to preserve the old task knowledge (Lee et al. 2017; Zenke, Poole, and Ganguli 2017; Liu et al. 2018).”
Curriculum-Meta Learning for Order-Robust Continual Relation Extraction (2021, methodology): “(i) Consolidation-based methods (Kirkpatrick et al. 2016; Zenke, Poole, and Ganguli 2017; Liu et al. 2018; Ritter, Botev, and Barber 2018) consolidate model parameters important to previous tasks and reduce their learning weights.”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, neutral): “Weight regularization has also been investigated (Aljundi et al., 2018; Kirkpatrick et al., 2017; Zenke et al., 2017; Liu et al., 2018).”
On Class Orderings for Incremental Learning (2020, methodology): “We consider disjoint classes between all tasks, C t ∩ C s = ∅ for t (cid:54) = s as in (Aljundi et al., 2017; Chaudhry et al., 2018; Dhar et al., 2019; Hou et al., 2019; Liu et al., 2018; Rebufﬁ et al., 2017; Yu et al., 2020).”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “Regularization strategies [36], [63], [64], [65], [66], [67], [68], [69], [70], [71] alleviate forgetting by regularization loss terms enabling the updated parameters of networks to retain past knowledge.”
Few-Shot Class-Incremental Learning (2020, background): “These works can be categorized into three types: (1) rehearsal approaches [24, 3, 37, 50, 46], (2) architectural approaches [27, 26, 1, 36, 47], and (3) regularization approaches [15, 49, 23, 18].”
Generative Feature Replay For Class-Incremental Learning (2020, methodology): “on new tasks so the network remains stable on previous ones [1, 19, 23, 24, 56].”