Conditional Channel Gated Networks for Task-Aware Continual Learning

By Davide Abati, J. Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, Babak Ehteshami Bejnordi. 2020

208 citations recorded by the source corpus.

Convolutional Neural Networks experience catastrophic forgetting when optimized on a sequence of learning problems: as they meet the objective of the current training examples, their performance on previous tasks drops drastically. In this work, we introduce a novel framework to tackle this problem with conditional computation. We equip each convolutional layer with task-specific gating modules, selecting which filters to apply on the given input. This way, we achieve two appealing properties. Firstly, the execution patterns of the gates allow to identify and protect important filters, ensuring no loss in the performance of the model for previously learned tasks. Secondly, by using a sparsity objective, we can promote the selection of a limited set of kernels, allowing to retain sufficient model capacity to digest new tasks. Existing solutions require, at test time, awareness of the task to which each example belongs to. This knowledge, however, may not be available in many practical scenarios. Therefore, we additionally introduce a task classifier that predicts the task label of each example, to deal with settings in which a task oracle is not available. We validate our proposal on four continual learning datasets. Results show that our model consistently outperforms existing methods both in the presence and the absence of a task oracle. Notably, on Split SVHN and Imagenet-50 datasets, our model yields up to 23.98% and 17.42% improvement in accuracy w.r.t. competing methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Mixture-of-Experts with Expert Choice Routing (2022, methodology): “Conditional convolution layers [1] with task-speciﬁc gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized.”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “For a regular network, the task-speciﬁc components could be parallel branches [233], [234], adaptive layers [58], [235], masks of intermediate features [186], [236], [237].”
Learnability and Algorithm for Continual Learning (2023, methodology): “CCG (Abati et al., 2020), Expert Gate (Aljundi et al., 2017), HyperNet (von Oswald et al., 2020) and PREnt (Henning et al., 2021) either build a separate network or use entropy to predict the task-id.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “CCG [63] builds a separate network to predict the task-id.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “While rehearsal strategies are by far the most frequently adopted thanks to their effectiveness and flexibility [26, 4], it is not uncommon to adopt solutions combining multiple approaches [32, 1, 15, 60, 18].”
Class Gradient Projection For Continual Learning (2022, methodology): “Many approaches have been proposed to deal with this dilemma, including i) regularization-based methods [2, 19], ii) memory-based methods [7, 27], and iii) expansion-based methods [1, 25].”
Anti-Retroactive Interference for Lifelong Learning (2022, methodology): “In [1], each convolutional layer is equipped with task-specific gating modules, selecting specific filters for a given task.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Architecture-based methods modify the network architecture dynamically to fit more tasks, by expanding the model by adding more weights [55, 52], or masking and allocating subnetworks to specific tasks [45], or by gating the parameters dynamically using a task identifier [1].”
Online Continual Learning with Contrastive Vision Transformer (2022, neutral): “As new tasks arrive, expansion-based methods dynamically expand networks and keep sub-networks related to previous tasks fixed [69,89,1,59,86,45,82,53].”
Novel Class Discovery without Forgetting (2022, background): “Dynamically expanding and parameter isolation methods [44,40,39,1,33] form a third class of methods to address forgetting.”
Incremental Task Learning with Incremental Rank Updates (2022, background): “[1] proposes task specific convolutional filter selection for continual learning.”
Transfer without Forgetting (2022, background): “While recent works provide a target expected activation ratio [1,58] as a countermeasure, we encourage the auxiliary modules to assign different propagation gating masks to different examples.”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “The exceptions infer the task identity typically by learning a routing classifier [1, 4], or estimating confidence scores [27].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Another research line [1, 44] explores the sparsity regularization for network parameters, which aims at activating as few neurons as possible for each task.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[164] proposed a gating module for each convolution layer to select a limited set of filters and protect the filters of the previous tasks from being unnecessarily updated.”