Ternary Feature Masks: zero-forgetting for task-incremental learning

By Marc Masana, Tinne Tuytelaars, Joost van de Weijer. 2021

35 citations recorded by the source corpus.

We propose an approach without any forgetting to continual learning for the task-aware regime, where at inference the task-label is known. By using ternary masks we can upgrade a model to new tasks, reusing knowledge from previous tasks while not forgetting anything about them. Using masks prevents both catastrophic forgetting and backward transfer. We argue – and show experimentally – that avoiding the former largely compensates for the lack of the latter, which is rarely observed in practice. In contrast to earlier works, our masks are applied to the features (activations) of each layer instead of the weights. This considerably reduces the number of mask parameters for each new task; with more than three orders of magnitude for most networks. The encoding of the ternary masks into two bits per feature creates very little overhead to the network, avoiding scalability issues. To allow already learned features to adapt to the current task without changing the behavior of these features for previous tasks, we introduce task-specific feature normalization. Extensive experiments on several finegrained datasets and ImageNet show that our method outperforms current state-of-the-art while reducing memory overhead in comparison to weight-based approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continually Learning Self-Supervised Representations with Projected Functional Regularization (2021, methodology): “Previous works following this strategy include DER [66], Piggyback [42], PackNet [43], DAN [54], HAT [56], Ternary Masks [45] and PathNet [21].”
Generalized Source-free Domain Adaptation (2021, background): “In continual learning literature the masking of weights [24, 25] and activations [1, 27, 33] has been studied.”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, methodology): “Some previous methods carefully select and assign a subset of weights in a large network to each task (Mallya & Lazebnik, 2018; Sokar et al., 2021; Fernando et al., 2017; Serra et al., 2018; Masana et al., 2021; Li et al., 2019; Yoon et al., 2018) or assign a task-specific network to each task (Rusu”