Adding New Tasks to a Single Network with Weight Trasformations using Binary Masks

By Massimiliano Mancini, Elisa Ricci, Barbara Caputo, Samuel Rota Bulò. 2018

56 citations recorded by the source corpus.

Visual recognition algorithms are required today to exhibit adaptive abilities. Given a deep model trained on a specific, given task, it would be highly desirable to be able to adapt incrementally to new tasks, preserving scalability as the number of new tasks increases, while at the same time avoiding catastrophic forgetting issues. Recent work has shown that masking the internal weights of a given original conv-net through learned binary variables is a promising strategy. We build upon this intuition and take into account more elaborated affine transformations of the convolutional weights that include learned binary masks. We show that with our generalization it is possible to achieve significantly higher levels of adaptation to new tasks, enabling the approach to compete with fine tuning strategies by requiring slightly more than 1 bit per network parameter per additional task. Experiments on two popular benchmarks showcase the power of our approach, that achieves the new state of the art on the Visual Decathlon Challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Typically, previous task parameters are kept fixed (Rusu et al., 2016; Mancini et al., 2018) or masked out (Serra et al., 2018; Mallya and Lazebnik, 2018).”
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning (2019, methodology): “To this end, [13, 14] learn to mask a pre-trained network without changing the weights of the base networks, whereas [29] (HAT) features binary mask-learning for the layer activations simultaneously to the training of the base network.”
Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting (2019, methodology): “Additionally, there are also methods (Mallya & Lazebnik, 2018; Rebuffi et al., 2017a; 2018; Mancini et al., 2018) that could learn multiple domains and completely avoid forgetting by adding a small portion of parameters while the original weights is kept fixed.”
How fine can fine-tuning be? Learning efficient language models (2020, methodology): “In contrast to commonly employed post-training sparsification methods which always incur performance degradation, our procedure of sparsifying pre-trained networks (similar to Mallya et al., 2018; Mancini et al., 2019) is by itself an optimization process that learns specific tasks.”