Incremental Learning Through Deep Adaptation

By Amir Rosenfeld, John K. Tsotsos. 2017

307 citations recorded by the source corpus.

Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added domain, typically as many as the original network. We propose a method called Deep Adaptation Modules (DAM) that constrains newly learned filters to be linear combinations of existing ones. DAMs precisely preserve performance on the original domain, require a fraction (typically 13 percent, dependent on network architecture) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3 percent of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, methodology): “In vision, convolutional adapter modules have been studied (Rebuffi et al., 2017; 2018; Rosenfeld & Tsotsos, 2018).”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Architecture-based methods allocate diﬀerent parameters or add new parameters for the new task, while sharing parameters across tasks to reduce the interference of previous tasks [3, 22, 27–29, 41, 42, 44, 53, 56, 60].”
Learning with Recoverable Forgetting (2022, neutral): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, methodology): “This parameter isolation is often achieved by growing new branches for tasks and freezing previous task parameters (Rusu et al., 2016; Li et al., 2021; Sahbi & Zhan, 2021; Yoon et al., 2018; Aljundi et al., 2017; Rosenfeld & Tsotsos, 2018).”
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches (2022, methodology): “Based on the task at hand, IL can be categorised into (a) Incremental domain learning [42], which aims at performing incremental domain adaptation.”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “Expansion-based methods (e.g., (Rusu et al., 2016; Li & Hoiem, 2017; Rosenfeld & Tsotsos, 2018; Hung et al., 2019; Yoon et al., 2017; Li et al., 2019; Veniat et al., 2020)) dynamically expand the network capacity to reduce the interference between the new tasks and the old ones.”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, background): “Additionally, Rosenfeld and Tsotsos (2018) proposed to optimize loss on the new task with representations learned from old tasks.”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “There are three main approaches to mitigating catastrophic forgetting [53]: 1) increasing model capacity to include new representations that will not interfere with old knowledge [67, 60, 58, 3, 57, 54, 46, 45, 65], 2) regularizing continual parameter updates such that parameters do not deviate too ”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Prominent works that follow this approach include Piggyback [29], PackNet [30], HAT [43], TFM [32], DAN [40], PathNet [14].”
A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix (2020, neutral): “…can cite the following: regularization-based methods Kirkpatrick et al. [2017], Zenke et al. [2017], memory-based and projection meth-ods [Chaudhry et al., 2018, Lopez-Paz and Ranzato, 2017, Farajtabar et al., 2020] or parameters isolations [Mallya and Lazebnik, 2018, Rosenfeld and Tsotsos, 2018].”
Meta-Consolidation for Continual Learning (2020, background): “Another kind of methods [51, 71, 68, 19, 67, 61, 86] attempt to expand the network to increase the capacity of the model, while learning new tasks.”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “More relevant to our work, in a multi-domain setting, a few approaches [49,50,53,37] utilize a pre-trained network that remains untouched and instead learn domain-specific components that adapt the behavior of the network to address the performance drop common in IL techniques.”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “Incremental learning is usually conducted under the task-incremental [72], [73] or the class-incre-mental learning scenarios [2], [70], [74], [75], [76], [77].”
Residual Continual Learning (2020, methodology): “In (Rosenfeld and Tsotsos 2018), newly added filters for a target task are learned in the form of linear combination with existing filters of source tasks.”
Compacting, Picking and Growing for Unforgetting Continual Learning (2019, background): “Dynamic architecture : Dynamic-architecture approaches [38, 20, 36, 29, 48] adapt the architecture with a sequence of tasks.”
IL2M: Class Incremental Learning With Dual Memory (2019, methodology): “Deep Adaptation Networks [25] are an alternative to fine tuning to fit models to new tasks.”
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning (2019, methodology): “Lately, a wealth of works dealing with catastrophic forgetting in context of DNNs have appeared in the literature, see e.g., [9, 34, 12, 29, 1, 24].”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “PackNet [56] PathNet [32] Piggyback [57] HAT [58] PNN [59] Expert Gate [7] RCL [60] DAN [19] Fig.”