Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

By Arun Mallya, Dillon Davis, Svetlana Lazebnik. 2018

791 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Efficiency & Compression

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “Within a ﬁxed network architecture, Piggyback [221], HAT [71], WSN [222] and H 2 [223] explicitly optimize a binary mask to select dedicated neurons or parameters for each task, with the masked regions of the old tasks (almost) frozen to prevent catastrophic forgetting.”
Weight Agnostic Neural Networks (2019, background): “Network Pruning By removing connections with small weight values from a trained neural network, pruning approaches [35, 39, 41, 58, 62, 66, 67, 69, 75] can produce sparse networks that keep only a small fraction of the connections, while maintaining similar performance on image classiﬁcation tasks…”
Long-Tailed Class Incremental Learning (2022, methodology): “Parameter-isolation methods increase model plasticity by adding more neurons, modules [31,33], branches [23] or masks [26,25,34].”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Architecture-based methods allocate diﬀerent parameters or add new parameters for the new task, while sharing parameters across tasks to reduce the interference of previous tasks [3, 22, 27–29, 41, 42, 44, 53, 56, 60].”
Online Continual Learning with Contrastive Vision Transformer (2022, neutral): “As new tasks arrive, expansion-based methods dynamically expand networks and keep sub-networks related to previous tasks fixed [69,89,1,59,86,45,82,53].”
Incremental Task Learning with Incremental Rank Updates (2022, background): “Piggyback [18] learns binary masks on an existing network to provide good performance on new tasks.”
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, methodology): “This is signiﬁcantly different than methods that utilize task information throughout the architecture, such as (Rusu et al., 2016; Mallya et al., 2018).”
Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “L G ] 1 6 Ju n 20 22 other hand, methods that do guarantee lack of forgetting, e.g. Rusu et al. (2016); Mallya & Lazebnik (2018); Mallya et al. (2018), operate in conditions that are far from real life — they assume that the task identity is known at inference, and hence we are allowed to adapt the…”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “However, they often introduce a substantial number of additional parameters and sometimes involve simplified assumption like known test time task identity [11, 35, 34], which falls into the setting of task-incremental learning.”
Class-Incremental Learning with Strong Pre-trained Models (2022, methodology): “On the other hand, methods based on dynamic models learn separate parameters for novel tasks, either by expanding the model [24], or introducing a parameter gating function [18,19].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, background): “Existing works in the literature can be grouped into three main categories: rehearsal-based methods [16], [30], [31], [32], [33], parameter isolation approaches [34], [35] and distillation-based techniques [14], [36], [37].”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “This is often accomplished either by progressively growing the model each time a new task is added [ 37 , 68 , 82 ], or maintaining a common backbone model which is adapted to each task separately using small task-specific adaptation blocks [41, 51, 66].”
Continual Prompt Tuning for Dialog State Tracking (2022, methodology): “nik, 2018), learning a parameter mask for each task individually (Mallya et al., 2018), etc.”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, methodology): “The main problem 135 CL attempts to address is catastrophic forgetting 136 (McCloskey and Cohen, 1989), i.e., the model for- 137 gets previous knowledge after learning new tasks.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “, 2020), or iteratively learn mask (Mallya et al., 2018; Serra et al., 2018) or prunes networks (Mallya and Lazebnik, 2018) for new tasks.”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, methodology): “First, architecture-based methods dynamically adjust the model architecture to learn new knowledge while preventing the forgetting of previously learned tasks (Chen et al., 2015; Rusu et al., 2016; Mallya et al., 2018).”
Learning with Selective Forgetting (2021, methodology): “Several strategies have been proposed, such as networks switching the nodes or branches to be used depending on the tasks [Mallya and Lazebnik, 2018; Mallya et al., 2018] or adding new nodes or branches every time a new task is learned [Rusu et al.”
FedSpeech: Federated Text-to-Speech with Continual Learning (2021, methodology): “As shown in Figure 1, we follow a sequential training setup, which is common in continual learning [Mallya and Lazebnik, 2018; Mallya et al., 2018; Hung et al., 2019].”