DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

By Arthur Douillard, Alexandre Ram'e, Guillaume Couairon, Matthieu Cord. 2021

468 citations recorded by the source corpus.

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an ex-pansion of the parameters can reduce catastrophic forget-ting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having neg-ligible memory and time overheads due to strict control of the expansion of the parameters. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet100 while having fewer parameters than concurrent dynamic frameworks.11Code is released at https://github.com/arthurdouillard/dytox.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “lel branches (ACL [107], ReduNet [459], EPIE-Net [104]), adaptive layers (GVCL [276], DyTox [101]), masks or mask”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “Similarly to SGD or the algorithm of Evron et al. (2022), in the worse case, this stochastic method forgets if it does not revisit past data, and even if [Y1 · · · YT ] is exactly of rank r; note that we did not give a formal proof for this claim, but it can be derived by combining the minimality of”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, background): “However, most of the state-of-the-art CL methods [11, 12, 20] are based on a strong assumption: the data is fully labeled.”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, methodology): “Finally, DyTox [14] learns a unified model with a parameter-isolation approach and dynamically expands the tokens processed by the last layer to mitigate forgetting.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “For instance, two recent prompting methods [14, 66] 5 aim at learning task/domain-speciﬁc prompts dependently across domains in an incremental update fashion.”
Online Continual Learning with Contrastive Vision Transformer (2022, background): “Since then, a lot of effort has been dedicated to improving Vision Transformers’ data efficiency and model efficiency [92,32,91,40], where an effective direction is to strategically integrate properties of convolution into the Transformer architecture [73,91,48,84,26,94,23,17].”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, methodology): “Recently, methods [9,28,40] successfully apply the dynamic architectures into class incremental learning where the task id is unavailable, showing their advantages over the single backbone methods.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “Among those recent works, only a few [30, 15] have applied the Transformers architecture to CL on image datasets.”
What matters when building vision-language models? (2024, unknown): “The number of queries (also referred to as latents) corresponds to the number of resulting visual tokens after the pooling (from one for classification Touvron et al. (2021) to several in Alayrac et al. (2022) or a growing number for continual learning Douillard et al. (2022)).”
Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection (2023, methodology): “…constrain learning on new classes and penalizing updating on weights [2,30] or gradients [42] while data regularization methods perform distillation [25] between the intermediate features [14,15,26,33] or attention maps [12] of the teacher model and the current student model to reduce forgetting.”
Deep Class-Incremental Learning: A Survey (2023, background): “In DyTox, only task tokens are expanded for each new task, which requires much less memory than saving the whole backbone.”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “Rather than storing exemplars, recent works [54, 50, 30, 12] find that saving backbones from the history pushes the performance by one step towards the upper bound.”
CoMFormer: Continual Learning in Semantic and Panoptic Segmentation (2022, methodology): “To the best of our knowledge, transformers have been evaluated only in continual learning for image classification [19, 31] and we are the first to use them in the continual segmentation task.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “A recent papers on natural language processing (NLP) indicate that SSL of large transformer models (Devlin et al., 2018; Reddy et al., 2021; Douillard et al., 2021) on vast amounts of pre-training data results in models that, without further training, generalize well to new tasks from few examples…”
Continual-T0: Progressively Instructing 50+ Tasks to Language Models Without Forgetting (2022, background): “(Douillard et al., 2021) proposed a a dynamic expansion of special tokens with a transformer architecture.”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, methodology): “As we can see, compared with using more experts, the task-specific routers contribute more to improve anti-forgetting and zero-shot transfer abilities.”
Convolutional Prompting meets Language Models for Continual Learning (2024, methodology): “(3) Rehearsal-based methods [3, 4, 12, 20– 22, 41, 52] store a few representative training samples from previous tasks in a buffer, which is then used for training alongside the current task.”
Class-Incremental Learning: A Survey (2023, methodology): “However, with the rapid development of the CIL field, many great works are emerging day by day, which substantially boost the performance of benchmark settings [15], [16], [17], [18].”
Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, background): “Recently, expandable networks [10, 17, 29, 30, 56, 64] show strong performance among other competitors.”