PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

By Arun Mallya, Svetlana Lazebnik. 2017

1,641 citations recorded by the source corpus.

This paper presents a method for adding multiple tasks to a single deep neural network while avoiding catastrophic forgetting. Inspired by network pruning techniques, we exploit redundancies in large deep networks to free up parameters that can then be employed to learn new tasks. By performing iterative pruning and network re-training, we are able to sequentially "pack" multiple tasks into a single network while ensuring minimal drop in performance and minimal storage overhead. Unlike prior work that uses proxy losses to maintain accuracy on older tasks, we always optimize for the task at hand. We perform extensive experiments on a variety of network architectures and large-scale datasets, and observe much better robustness against catastrophic forgetting than prior work. In particular, we are able to add three fine-grained classification tasks to a single ImageNet-trained VGG-16 network and achieve accuracies close to those of separately trained networks for each task.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “, 2017), and packing new tasks into already trained networks (Mallya and Lazebnik, 2017).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “PackNet [224], UCL [115], CLNP [225] and AGS-CL [116] explicitly identify the important neurons or parameters for the current task and then release the unimportant parts to the following tasks, which can be achieved by iterative pruning [224], activation value [116], [225], [226], uncertainty…”
Overcoming catastrophic forgetting with hard attention to the task (2018, methodology): “Comparing these numbers with the compression rates used by PackNet (50 or 25%), we see that HAT generally uses a much more compact model.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Parameter-isolation methods train and protect a sub-network for each task (Mallya & Lazebnik, 2017; Abati et al., 2020; von Oswald et al., 2020; Rajasegaran et al., 2020b; Hung et al., 2019; Henning et al., 2021).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “…2021; Kim et al., 2022; Zhang et al., 2022; Wang et al., 2022a; Saha & Roy, 2023), expansion-based methods (Rusu et al., 2016; Yoon et al., 2018; Mallya & Lazebnik, 2018; Li et al., 2019; Hung et al., 2019; Ramesh & Chaudhari, 2022; Douillard et al., 2022), or combinations thereof (Yan et al.,…”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, neutral): “Comparison with evolving architectures: Similar to progressive networks (e.g. PNN, CPG, PAE), dynamic sparse networks (e.g. CLIP, NISPA, PackNet) reduce task interference by learning a nonoverlapping task-specific sparse architecture within a fixed capacity model.”
A Theoretical Study on Solving Continual Learning (2022, background): “Parameter isolation is yet another popular approach, which makes different subsets (which may overlap) of the network parameters dedicated to different tasks using masks [3, 52, 53, 4, 54].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “Differently, structural methods purposefully organize the allocation of model capacity to prevent interference and facilitate parameter sharing [1, 51, 66, 37].”
Long-Tailed Class Incremental Learning (2022, methodology): “Parameter-isolation methods increase model plasticity by adding more neurons, modules [31,33], branches [23] or masks [26,25,34].”
Exploring Example Influence in Continual Learning (2022, neutral): “The parameter isolation based methods [14, 26] generate task-speciﬁc parameter expansion or sub-branch.”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Parameter isolation methods [3, 29, 40, 42] are based on dynamically allocating a set of parameters for each task so that the training does not interfere with each other.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “These method can be roughly divided into four categories: architecture-based methods expand the network or allocate new neurons for new tasks [13,28,42,60]; replayedbased methods interleave old data with current data by storing historical data in a buffer or generating virtual old data [4,8,12,39,40”
Incremental Task Learning with Incremental Rank Updates (2022, background): “We can observe that our approach outperforms HAT and PackNet for RMNIST, S-CIAR100 and S-miniImageNet with a significantly smaller number of parameters.”
Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “L G ] 1 6 Ju n 20 22 other hand, methods that do guarantee lack of forgetting, e.g. Rusu et al. (2016); Mallya & Lazebnik (2018); Mallya et al. (2018), operate in conditions that are far from real life — they assume that the task identity is known at inference, and hence we are allowed to adapt the…”
Transfer without Forgetting (2022, background): “Architectural approaches dedicate distinct sets of parameters to each task, often resorting to network expansion as new tasks arrive [56,38,58].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “These methods can be further categorized as expanding the model [51, 65, 26, 31, 59], or dividing the model [35, 52, 58, 17, 11].”