Continual Learning With Extended Kronecker-Factored Approximate Curvature

By Janghyeon Lee, Hyeong Gwon Hong, Donggyu Joo, Junmo Kim. 2020

72 citations recorded by the source corpus.

We propose a quadratic penalty method for continual learning of neural networks that contain batch normalization (BN) layers. The Hessian of a loss function represents the curvature of the quadratic penalty function, and a Kronecker-factored approximate curvature (K-FAC) is used widely to practically compute the Hessian of a neural network. However, the approximation is not valid if there is dependence between examples, typically caused by BN layers in deep network architectures. We extend the K-FAC method so that the inter-example relations are taken into account and the Hessian of deep neural networks can be properly approximated under practical assumptions. We also propose a method of weight merging and reparameterization to properly handle statistical parameters of BN, which plays a critical role for continual learning with BN, and a method that selects hyperparameters without source task data. Our method shows better performance than baselines in the permuted MNIST task with BN layers and in sequential learning from the ImageNet classification task to fine-grained classification tasks with ResNet-50, without any explicit or implicit use of source task data for hyperparameter selection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Geophysics, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Da-Wei Zhou
5. Joost van de Weijer
6. Hang Su
7. Liyuan Wang
8. Xingxing Zhang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. Joost van de Weijer
3. De-chuan Zhan
4. Dacheng Tao
5. Jun Zhu
6. Ziwei Liu
7. Luc Van Gool
8. Bernt Schiele

In Optimization & Training

1. Jun Zhu
2. Liyuan Wang
3. Xingxing Zhang
4. Sungmin Cha
5. Taesup Moon
6. Hang Su
7. Hyounguk Shon
8. Yajing Kong

In Meta & Continual Learning

1. Joost van de Weijer
2. Han-Jia Ye
3. De-chuan Zhan
4. Liyuan Wang
5. Da-Wei Zhou
6. Xingxing Zhang
7. Jun Zhu
8. Hang Su

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “XK-FAC [109] further considers the inter-example relations in approximating the FIM to better accommodate batch normalization.”
• DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Naturally, existing works have focused on efficient representation of the matrix via diagonal approximation [18, 25] or Kronecker factorization [22, 39].”
• Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Moreover, with new tasks occurring, the number of previous tasks increases, resulting in a greater impact of previous tasks on the final performance than the current task [19, 21, 58].”
• DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “Weights constraints as EWC [14] or second-order optimization [17] have similar motivations.”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[28] pointed out that the current usage of the Hessian matrix approximation as the curvature of the quadratic penalty function is not effective for networks containing batch normalization layers.”
• MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, neutral): “Recently, there has been a large body of research in incremental learning [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43].”
• Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, methodology): “[44] propose an extension of the Kronecker factorization technique for block-diagonal approximation of the Fisher Information Matrix.”
• Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping (2021, methodology): “(Lee et al., 2020) does employ a Fuzzy Testing scenario for Split-MNIST in which there are transition phases between tasks where the amount of new data increases linearly in each batch.”
• Deep Class-Incremental Learning: A Survey (2023, methodology): “K-FAC [84] extends the Fisher information matrix approximation with the Kronecker factorization technique.”
• Task-Balanced Batch Normalization for Exemplar-based Class-Incremental Learning (2022, background): “Recently, appropriate use of BN in more diverse tasks, such as meta-learng (Bronskill et al., 2020), task-incremental learning (Lee et al., 2020) and on-line class-incremental learning (Anonymous, 2022), have been discussed.”
• Class-Incremental Learning: A Survey (2023, methodology): “…[66], [67], [68] Prompt Expansion [18], [69], [70], [71], [72] § 3.4: Parameter Regularization [33], [73], [74], [75], [76], [77], [78], [79], [80] § 3.5 Knowledge Distillation Logit Distillation [81], [82], [83], [84], [85], [86], [87], [88] Feature Distillation [89], [90], [91], [92], [93],…”