Self-Net: Lifelong Learning via Continual Self-Modeling

By Jaya Krishna Mandivarapu, Blake Camp, Rolando Estrada. 2018

18 citations recorded by the source corpus.

Learning a set of tasks over time, also known as continual learning (CL), is one of the most challenging problems in artificial intelligence. While recent approaches achieve some degree of CL in deep neural networks, they either (1) store a new network (or an equivalent number of parameters) for each new task, (2) store training data from previous tasks, or (3) restrict the network's ability to learn new tasks. To address these issues, we propose a novel framework, Self-Net, that uses an autoencoder to learn a set of low-dimensional representations of the weights learned for different tasks. We demonstrate that these low-dimensional vectors can then be used to generate high-fidelity recollections of the original weights. Self-Net can incorporate new tasks over time with little retraining, minimal loss in performance for older tasks, and without storing prior training data. We show that our technique achieves over 10X storage compression in a continual fashion, and that it outperforms state-of-the-art approaches on numerous datasets, including continual versions of MNIST, CIFAR10, CIFAR100, Atari, and task-incremental CORe50. To the best of our knowledge, we are the first to use autoencoders to sequentially encode sets of network weights to enable continual learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Meta-Consolidation for Continual Learning (2020, background): “[52] auto-encode the weights in a continual setting, but the size of”
Continual learning with hypernetworks (2019, background): “, 2019) or distills parameters onto a contractive auto-encoding model store (Camp et al., 2018).”
Continual Learning with Deep Artificial Neurons (2020, background): “Broadly speaking, 4 categories of techniques have emerged to combat catastrophic forgetting: (1) regularization-based approaches [4, 22, 24], (2) knowledge-compression, or capacity expansion [20, 23, 28, 38], (3) experience replay and/or external memory slots [9, 29, 36, 40], and (4) metalearned rep”