Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process

By Fei Ye, Adrian G. Bors. 2021

26 citations recorded by the source corpus.

Recent research efforts in lifelong learning propose to grow a mixture of models to adapt to an increasing number of tasks. The proposed methodology shows promising results in overcoming catastrophic forgetting. However, the theory behind these successful models is still not well understood. In this paper, we perform the theoretical analysis for lifelong learning models by deriving the risk bounds based on the discrepancy distance between the probabilistic representation of data generated by the model and that corresponding to the target dataset. Inspired by the theoretical analysis, we introduce a new lifelong learning approach, namely the Lifelong Infinite Mixture (LIMix) model, which can automatically expand its network architectures or choose an appropriate component to adapt its parameters for learning a new task, while preserving its previously learnt information. We propose to incorporate the knowledge by means of Dirichlet processes by using a gating mechanism which computes the dependence between the knowledge learnt previously and stored in each component, and a new set of data. Besides, we train a compact Student model which can accumulate cross-domain representations over time and make quick inferences. The code is available at https://github.com/dtuzi123/Lifelong-infinite-mixture-model.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Ye
2. Adrian G. Bors

Citations in the corpus

• Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, methodology): “This issue is relieved by using ENA (Lee et al. 2020), inspired by a network expansion mechanism (Rao et al. 2019), or by employing a combination between ENA and GR mechanisms (Ye and Bors 2021f,e).”
• Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, background): “Different from prior theoretical analysis works [23, 21], the proposed theoretical analysis can be used in both the OCL and in a general continual learning setting where the task information is provided.”
• Learning an Evolved Mixture Model for Task-Free Continual Learning (2022, background): “The dynamic models usually would add new layers with hidden nodes [15] within a single neural network or a task-specific module into a mixture system [10, 16, 17].”