Ternary Feature Masks: continual learning without any forgetting

By Marc Masana, Tinne Tuytelaars, Joost van de Weijer. 2020

30 citations recorded by the source corpus.

In this paper, we propose an approach without any forgetting to continual learning for the task-aware regime, where at inference the task-label is known. By using ternary masks we can upgrade a model to new tasks, reusing knowledge from previous tasks while not forgetting anything about them. Using masks prevents both catastrophic forgetting and backward transfer. We argue -- and show experimentally -- that avoiding the former largely compensates for the lack of the latter, which is rarely observed in practice. In contrast to earlier works, our masks are applied to the features (activations) of each layer instead of the weights. This considerably reduces the number of mask parameters to be added for each new task; with more than three orders of magnitude for most networks. The encoding of the ternary masks into two bits per feature creates very little overhead to the network, avoiding scalability issues. Our masks do not permit any changes to features which are used by previous tasks. As this may be too restrictive to allow learning of new tasks, we add task-specific feature normalization. This way, already learned features can adapt to the current task without changing the behavior of these features for previous tasks. Extensive experiments on several finegrained datasets and ImageNet show that our method outperforms current state-of-the-art while reducing memory overhead in comparison to weight-based approaches.

Subfields: Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

ACAE-REMIND for Online Continual Learning with Compressed Feature Replay (2021, methodology): “This method is further developed to the case were no forgetting is allowed in [26].”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Prominent works that follow this approach include Piggyback [29], PackNet [30], HAT [43], TFM [32], DAN [40], PathNet [14].”
Online Continual Learning under Extreme Memory Constraints (2020, methodology): “The bounding boxes, provided with the data, are used to crop the images which are then resized as in [2] to 224x224 pixels to fit them into a ResNet18 [1].”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “, 2018) and feature masking (Serra et al., 2018; Masana et al., 2020) or PiggyBack (Mallya et al.”
On Class Orderings for Incremental Learning (2020, neutral): “In (Masana et al., 2020) and (De Lange et al., 2019), the authors touch the subject of class ordering, showing that some methods report different results based on different class orderings.”
Generative Feature Replay For Class-Incremental Learning (2020, background): “Piggyback [28] and Packnet [29] and learn a separate mask for each task, while HAT [45] and Ternary Feature Masks [30] learn a mask on the activations instead of for each parameter.”
Semantic Drift Compensation for Class-Incremental Learning (2020, background): “Many of the early works in continual learning considered a task-incremental learning (task-IL) scenario [40], in which the network has access to the task-ID at inference time [1, 17, 20, 26, 36].”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “This forces several forward passes with different masks, which makes such methods effective for task-aware evaluation, but impractical for task-agnostic settings [13], [64].”
Efficient Feature Transformations for Discriminative and Generative Continual Learning (2021, methodology): “A number of previous works [66, 59, 43, 57, 8, 45, 29, 30, 51, 58] have proposed methods for expanding a neural network to learn a sequence of tasks.”
A Continual Learning Survey: Defying Forgetting in Classification Tasks (2019, methodology): “Besides the original results by Serra et al. [63], our results for Tiny Imagenet, and difﬁculties for other setups correspond to those found in other work [81].”