Routing Networks with Co-training for Continual Learning

By Mark Collier, Efi Kokiopoulou, Andrea Gesmundo, Jesse Berent. 2020

15 citations recorded by the source corpus.

The core challenge with continual learning is catastrophic forgetting, the phenomenon that when neural networks are trained on a sequence of tasks they rapidly forget previously learned tasks. It has been observed that catastrophic forgetting is most severe when tasks are dissimilar to each other. We propose the use of sparse routing networks for continual learning. For each input, these network architectures activate a different path through a network of experts. Routing networks have been shown to learn to route similar tasks to overlapping sets of experts and dissimilar tasks to disjoint sets of experts. In the continual learning context this behaviour is desirable as it minimizes interference between dissimilar tasks while allowing positive transfer between related tasks. In practice, we find it is necessary to develop a new training method for routing networks, which we call co-training which avoids poorly initialized experts when new tasks are presented. When combined with a small episodic memory replay buffer, sparse routing networks with co-training outperform densely connected networks on the MNIST-Permutations and MNIST-Rotations benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Neil Houlsby
4. Xingxing Zhang
5. Basil Mustafa
6. Liyuan Wang
7. Matthieu Cord
8. Joan Puigcerver

In Computer Vision and Pattern Recognition

1. Jun Zhu
2. Hang Su
3. Neil Houlsby
4. Matthieu Cord
5. Liyuan Wang
6. Basil Mustafa
7. Xingxing Zhang
8. Joan Puigcerver

In Meta & Continual Learning

1. Liyuan Wang
2. Xingxing Zhang
3. Hang Su
4. Jun Zhu
5. Arthur Douillard
6. Matthieu Cord
7. Guillaume Couairon
8. Alexandre Ram'e

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Also, the function of task-identity prediction can be explicitly learned from incremental tasks, using other continual learning strategies to mitigate catastrophic forgetting [1], [79], [107], [160], [189], [221], [294].”
• DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “Indeed, aiming for wider minima is particularly important in continual learning [14, 26].”
• Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, background): “These models however have other benefits; for example, the sparsity protects against catastrophic forgetting in continual learning [5] and can improve performance for multitask learning [6] by offering a convenient inductive bias.”