Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector

By Guoliang Lin, H. Chu, Hanjiang Lai. 2021

77 citations recorded by the source corpus.

Plasticity-stability dilemma is a main problem for incremental learning, where plasticity is referring to the ability to learn new knowledge, and stability retains the knowledge of previous tasks. Many methods tackle this problem by storing previous samples, while in some applications, training data from previous tasks cannot be legally stored. In this work, we propose to employ mode connectivity in loss landscapes to achieve better plasticity-stability trade-off without any previous samples. We give an analysis of why and how to connect two independently optimized optima of networks, null-space projection for previous tasks and simple SGD for the current task, can attain a meaningful balance between preserving already learned knowledge and granting sufficient flexibility for learning a new task. This analysis of mode connectivity also provides us a new perspective and technology to control the trade-off between plasticity and stability. We evaluate the proposed method on several benchmark datasets. The results indicate our simple method can achieve notable improvement, and perform well on both the past and current tasks. On 10-split-CIFAR-100 task, our method achieves 79.79% accuracy, which is 6.02% higher. Our method also achieves 6.33% higher accuracy on TinyImageNet. Code is available at https://github.com/lingl1024/Connector.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “…of sequentially arrived tasks within the inner loop, and exploitation of mode connectivity and ﬂat minima in loss landscape (adapted from [86], [113]). θ ∗ A , θ ∗ B and θ ∗ A,B are desirable solutions for task A , task B and both of them, respectively. and MER [131] constrain parameter…”
Continual Self-Supervised Learning: Towards Universal Multi-Modal Medical Data Representation Learning (2023, unknown): “These CL techniques can be summarized in five main categories, including regularization-based approach [25], rehearsal-based approach [5, 35], optimization-based approach [28], representation-based approach [14], and architecture-based approach [32].”