SPACE: Structured Compression and Sharing of Representational Space for Continual Learning

By Gobinda Saha, Isha Garg, Aayush Ankit, Kaushik Roy. 2020

26 citations recorded by the source corpus.

Humans learn incrementally from sequential experiences throughout their lives, which has proven hard to emulate in artificial neural networks. Incrementally learning tasks causes neural networks to overwrite relevant information learned about older tasks, resulting in ‘Catastrophic Forgetting’. Efforts to overcome this phenomenon often utilize resources poorly, for instance, by growing the network architecture or needing to save parametric importance scores, or violate data privacy between tasks. To tackle this, we propose SPACE, an algorithm that enables a network to learn continually and efficiently by partitioning the learnt space into a Core space, that serves as the condensed knowledge base over previously learned tasks, and a Residual space, which is akin to a scratch space for learning the current task. After learning each task, the Residual is analyzed for redundancy, both within itself and with the learnt Core space. A minimal number of extra dimensions required to explain the current task are added to the Core space and the remaining Residual is freed up for learning the next task. We evaluate our algorithm on P-MNIST, CIFAR and a sequence of 8 different datasets, and achieve comparable accuracy to the state-of-the-art methods while overcoming catastrophic forgetting. Additionally, our algorithm is well suited for practical use. The partitioning algorithm analyzes all layers in one shot, ensuring scalability to deeper networks. Moreover, the analysis of dimensions translates to filter-level sparsity, and the structured nature of the resulting architecture gives us up to 5x improvement in energy efficiency during task inference over the current state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Gobinda Saha
2. Kaushik Roy
3. Yuke Zhu
4. Isha Garg
5. Bo Liu
6. Yihao Feng
7. Peter Stone
8. Yifeng Zhu

In Computer Vision and Pattern Recognition

1. Kaushik Roy
2. Gobinda Saha
3. Peter Stone
4. Isha Garg
5. Bo Liu
6. Qian Liu
7. Yifeng Zhu
8. Chongkai Gao

Citations in the corpus

• Gradient Projection Memory for Continual Learning (2021, methodology): “Instead we propose an approach that, after learning each task, partitions the entire gradient space of the weights into two orthogonal subspaces: Core Gradient Space (CGS) and Residual Gradient Space (RGS) (Saha et al., 2020).”
• Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, neutral): “Approaches include regularization penalties [50, 51, 52, 53, 54], meta learning [55], model compression [56, 57, 58], or knowledge distillation [5, 7, 8, 9].”
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, background): “Although some recent works explore the combination of different approaches [2, 29, 54] or new strategies [74, 57, 14], our benchmark aims to provide an in-depth analysis of these three basic lifelong learning directions to reveal their pros and cons on robot learning tasks.”
• Saliency Guided Experience Packing for Replay in Continual Learning (2021, methodology): “2018) under this category expand the network for accommodating new tasks, whereas in other methods (Mallya and Lazebnik 2018; Serrà et al. 2018; Saha et al. 2020) a task-specific subnetwork is selected by masking out the parameters.”