GROWN: GRow Only When Necessary for Continual Learning

By Li Yang, Sen Lin, Junshan Zhang, Deliang Fan. 2021

15 citations recorded by the source corpus.

Catastrophic forgetting is a notorious issue in deep learning, referring to the fact that Deep Neural Networks (DNN) could forget the knowledge about earlier tasks when learning new tasks. To address this issue, continual learning has been developed to learn new tasks sequentially and perform knowledge transfer from the old tasks to the new ones without forgetting. While recent structure-based learning methods show the capability of alleviating the forgetting problem, these methods start from a redundant full-size network and require a complex learning process to gradually grow-and-prune or search the network structure for each task, which is inefficient. To address this problem and enable efficient network expansion for new tasks, we first develop a learnable sparse growth method eliminating the additional pruning/searching step in previous structure-based methods. Building on this learnable sparse growth method, we then propose GROWN, a novel end-to-end continual learning framework to dynamically grow the model only when necessary. Different from all previous structure-based methods, GROWN starts from a small seed network, instead of a full-sized one. We validate GROWN on multiple datasets against state-of-the-art methods, which shows superior performance in both accuracy and model size. For example, we achieve 1.0\% accuracy gain on average compared to the current SOTA results on CIFAR-100 Superclass 20 tasks setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Junshan Zhang
2. Sen Lin
3. Li Yang
4. Deliang Fan

In Computer Vision and Pattern Recognition

1. Sen Lin
2. Li Yang
3. Deliang Fan
4. Junshan Zhang

Citations in the corpus

• Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, background): “, [24, 16, 31, 30, 26, 29]) allocate different model parameters to different tasks, fix the parameters for old tasks by masking them out during the new task learning and expand the network when needed.”