Model Zoo: A Growing Brain That Learns Continually

By Rahul Ramesh, Pratik Chaudhari. 2021

82 citations recorded by the source corpus.

This paper argues that continual learning methods can benefit by splitting the capacity of the learner across multiple models. We use statistical learning theory and experimental analysis to show how multiple tasks can interact with each other in a non-trivial fashion when a single model is trained on them. The generalization error on a particular task can improve when it is trained with synergistic tasks, but can also deteriorate when trained with competing tasks. This theory motivates our method named Model Zoo which, inspired from the boosting literature, grows an ensemble of small models, each of which is trained during one episode of continual learning. We demonstrate that Model Zoo obtains large gains in accuracy on a variety of continual learning benchmark problems. Code is available at https://github.com/grasp-lyrl/modelzoo_continual.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The generalization error for each task can improve with synergistic tasks but deteriorate with competing tasks [68], where learning all tasks equally in a shared solution tends to compromise each task in performance [68], [69].”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “…Saha & Roy, 2023), expansion-based methods (Rusu et al., 2016; Yoon et al., 2018; Mallya & Lazebnik, 2018; Li et al., 2019; Hung et al., 2019; Ramesh & Chaudhari, 2022; Douillard et al., 2022), or combinations thereof (Yan et al., 2021; Xie et al., 2022; Wang et al., 2022b; Frascaroli et…”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “[35] proposed a model zoo that incrementally adds sub-networks to learn new tasks, which had to store a large amount of old training samples.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “A distinct method splits the capacity of the model across tasks selected via boosting, and adds one such model for every new task (Ramesh and Chaudhari, 2022).”
Provable and Efficient Continual Representation Learning (2022, background): “Recent work [67] explores statistical challenges associated with continual learning in terms of the relatedness across tasks through replay-based strategies.”
Three types of incremental learning (2022, unknown): “13 for examples from the literature).”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “These include replay-based methods [30, 207, 213], parameter regularization [4, 113, 196], and model architecture expansion [191, 236].”