Continual Normalization: Rethinking Batch Normalization for Online Continual Learning

By Quang Pham, Chenghao Liu, Steven C. H. Hoi. 2022

76 citations recorded by the source corpus.

Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation is available at \url{https://github.com/phquang/Continual-Normalization}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Batch Normalization (BN) layers [179] tend to introduce biased moments towards the current task, resulting in sub-optimal performance of the previous tasks [337].”
Architecture Matters in Continual Learning (2022, background): “Similar to the results in Mirzadeh et al. (2020); Pham et al. (2022), we found the BN to facilitate the learning accuracy in Split CIFAR-100 and split ImageNet-1K ( cf. Tab.”
Class-Incremental Learning: A Survey (2023, background): “[ 196] suggests that batch normalization layers are biased when sequentially trained, which triggers the different activation of old and new classes.”
Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, methodology): “Model rectification-based methods [5, 43, 47, 63, 67, 73] correct the inductive bias of incremental models for unbiased prediction.”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, methodology): “According to the access to task identity during the training and inference phase, the research is often categorized into specific problems, such as task-incremental [34, 8], class-incremental [4, 25, 27, 37, 23, 3, 19, 46], and task-free continual learning [1, 21, 35, 11].”
Deep Class-Incremental Learning: A Survey (2023, background): “[172] finds that the batch normalization layers [173] are biased in CIL and proposes to re-normalize the layer outputs.”