Piggyback: Adding Multiple Tasks to a Single, Fixed Network by Learning to Mask

By Arun Mallya, Svetlana Lazebnik. 2018

36 citations recorded by the source corpus.

This work presents a method for adding multiple tasks to a single, fixed deep neural network without affecting performance on already learned tasks. By building upon concepts from network quantization and sparsification, we learn binary masks that "piggyback", or are applied to an existing network to provide good performance on a new task. These masks are learned in an end-to-end differentiable fashion, and incur a low overhead of 1 bit per network parameter, per task. Even though the underlying network is fixed, the ability to mask certain weights allows for the learning of a large number of filters. We show improved performance on a variety of classification tasks, including those with large domain shifts from the natural images of ImageNet. Unlike prior work, we can augment the capabilities of a network without suffering from catastrophic forgetting or competition between tasks, while incurring the least overhead per added task. We demonstrate the applicability of our method to multiple architectures, and obtain accuracies comparable with individual networks trained per task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning for Robotics (2019, methodology): “The way weight freezing is implemented in PackNet (Mallya and Lazebnik, 2017), Piggypack (Mallya and Lazebnik, 2018) or HAT (Serra et al., 2018) is by defining a special mask for each task that is used to both protect weights when new tasks are learned, and to define which weights to use at…”
Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data (2018, background): “The proposed approaches in [34] and [35] try to learn a mask, which marks important neurons for the old tasks.”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “Piggyback [22] first learns a general source task and then tries to adapt the learned network to more specific transfer tasks by learning a separate binary mask on its weights for each specific task.”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “We also consider a relaxed (soft) version of movement pruning based on the binary mask function described by Mallya and Lazebnik [2018].”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “We show the results of SpotTune and the baselines on the Visual Decathlon Challenge in Table 3.”
Adding New Tasks to a Single Network with Weight Trasformations using Binary Masks (2018, methodology): “A more compact and effective solution is [25], where separate binary masks are learned for each novel task and multiplied to the original network weights.”
Side-Tuning: Network Adaptation via Additive Side Networks (2019, methodology): “multiplication can be thought of as a measure of agreement among sensors, and multiplication for fusion has been explored in [3, 16].”
Representation Similarity Analysis for Efficient Task Taxonomy & Transfer Learning (2019, background): “It has been noted in multiple works [24, 33, 28], that the initialization plays a signiﬁcant role in performance in transfer learning.”