Adaptive Orthogonal Projection for Batch and Online Continual Learning
By Yiduo Guo, Wenpeng Hu, Dongyan Zhao, Bing Liu. 2022
57 citations recorded by the source corpus.
Catastrophic forgetting is a key obstacle to continual learning. One of the state-of-the-art approaches is orthogonal projection. The idea of this approach is to learn each task by updating the network parameters or weights only in the direction orthogonal to the subspace spanned by all previous task inputs. This ensures no interference with tasks that have been learned. The system OWM that uses the idea performs very well against other state-of-the-art systems. In this paper, we first discuss an issue that we discovered in the mathematical derivation of this approach and then propose a novel method, called AOP (Adaptive Orthogonal Projection), to resolve it, which results in significant accuracy gains in empirical evaluations in both the batch and online continual learning settings without saving any previous training data as in replay-based methods.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Optimization & Training, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Radiology, Nuclear Medicine and Imaging
In Optimization & Training
In Meta & Continual Learning
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “a pre-trained backbone for orthogonal gradient projection (AOP [144]) or feature replay (CV [159]).”
- The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “While Guo et al. (2022) showed that updating λ in a certain way at every batch improves the empirical performance, inverting X(j)1 (X (j) 1 ) > + λI might not be necessary if the goal is to compute (approximate) the nullspace of X(j)1 , which can be done via SVD.”
- A Theoretical Study on Solving Continual Learning (2022, background): “[50] Yiduo Guo, Wenpeng Hu, Dongyan Zhao, and Bing Liu.”
- Continual Learning Based on OOD Detection and Task Masking (2022, background): “, orthogonal projection [19, 32, 74], a network of experts [3] and generalized CL [47], etc.”
- Incremental Nuclei Segmentation from Histopathological Images via Future-class Awareness and Compatibility-inspired Distillation (2024, background): “The constraints can be applied on features [10, 12, 18], the output logits [4, 11, 21], the weights [1, 2, 19], or the gradients [6, 15].”
- AI Autonomy: Self-initiated Open-world Continual Learning and Adaptation (2022, background): “When the tasks are dissimilar overcoming CF is the key (Chen and Liu 2018; Guo et al. 2022).”
- Class-Incremental Learning: A Survey (2023, methodology): “In the following sections, we will discuss CIL methods from these aspects.”
- Dealing with Cross-Task Class Discrimination in Online Continual Learning (2023, background): “OWM [51], OGD [14], and AOP [15] avoid CF by projecting the gradients on the new task onto an orthogonal subspace of old tasks.”
- Online Continual Learning through Mutual Information Maximization (2022, background): “Online CL meth-ods are mainly based on replay except AOP (Guo et al., 2022), which uses a pre-trained model.”