Lifelong Teacher-Student Network Learning
By Fei Ye, Adrian G. Bors. 2021
57 citations recorded by the source corpus.
A unique cognitive capability of humans consists in their ability to acquire new knowledge and skills from a sequence of experiences. Meanwhile, artificial intelligence systems are good at learning only the last given task without being able to remember the databases learnt in the past. We propose a novel lifelong learning methodology by employing a Teacher-Student network framework. While the Student module is trained with a new given database, the Teacher module would remind the Student about the information learnt in the past. The Teacher, implemented by a Generative Adversarial Network (GAN), is trained to preserve and replay past knowledge corresponding to the probabilistic representations of previously learnt databases. Meanwhile, the Student module is implemented by a Variational Autoencoder (VAE) which infers its latent variable representation from both the output of the Teacher module as well as from the newly available database. Moreover, the Student module is trained to capture both continuous and discrete underlying data representations across different domains. The proposed lifelong learning framework is applied in supervised, semi-supervised and unsupervised training.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, neutral): “Another kind of approaches for continual learning focuses on the memory system, which usually employs a small memory buffer [3, 5, 6, 28, 36, 44, 59] for storing some past data or trains a generator to provide the replay samples when learning new tasks [38, 43, 46, 47, 58, 57, 52].”
- Learning an Evolved Mixture Model for Task-Free Continual Learning (2022, background): “branches: regularization-based [8], generative replay mechanism [4, 9] and dynamic expansion approaches [10, 11].”
- Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process (2021, background): “Memory replay approaches would either use a generator [1, 34, 33, 41, 56, 51, 52, 55, 57] or a memory buffer [2, 4, 46] as a replay mechanism generating data, which is statistically consistent to the previously learnt knowledge.”
- Lifelong Mixture of Variational Autoencoders (2021, background): “Recently, there have been some attempts to learn crossdomain representations under the lifelong learning by introducing an environment-dependent mask that specifies a subset of generative factors [16], by proposing a Teacher-Student lifelong learning framework [15], [36] or a hybrid model [37] of Ge”
- Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, methodology): “We also compare with LIMix (Ye and Bors 2021e) and implement CN-DPM (Lee et al. 2020) with the optimal setting, namely CN-DPM* (See details in Appendix-L.1 from SM1).”
- Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, background): “Other approaches would employ a small buffer to store a few past data [3,10,53] or would train a generator as a generative replay network that provides pseudo data samples for the future task learning [2,44,45,49,57,58,59,66,69].”