Supermasks in Superposition

By Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi. 2020

352 citations recorded by the source corpus.

We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance. If task identity is given at test time, the correct subnetwork can be retrieved with minimal memory usage. If not provided, SupSup can infer the task using gradient-based optimization to find a linear superposition of learned supermasks which minimizes the output entropy. In practice we find that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks. We also showcase two promising extensions. First, SupSup models can be trained entirely without task identity information, as they may detect when they are uncertain about new data and allocate an additional supermask for the new training distribution. Finally the entire, growing set of supermasks can be stored in a constant-sized reservoir by implicitly storing them as attractors in a fixed-sized Hopfield network.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Efficiency & Compression

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “Within a fixed network architecture, Piggyback [285], HAT [375], SupSup [451], MEAT [466], WSN [198] and H(2) [189] explicitly optimize a binary mask to select dedicated neurons or parameters for each task, with the masked regions of the old tasks (almost) frozen to prevent catastrophic forgetting.”
Learnability and Algorithm for Continual Learning (2023, methodology): “This is a valid assumption as existing algorithms (Serrà et al., 2018; Wortsman et al., 2020) achieve little or no forgetting.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “For the backbone structure, we follow [4, 26, 34].”
Incremental Task Learning with Incremental Rank Updates (2022, background): “SupSup [39] also finds masks in order to assign different subsets of the weights for different tasks.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “Most current applications focus on learning each single task with a sub-network [39,2,49], which can be seen as a special case of dynamic architecture.”
Continual Learning with Guarantees via Weight Interval Constraints (2022, methodology): “Alternatively, a static architecture can be reused with iterative pruning proposed by PackNet (Mallya & Lazebnik, 2018) or by using Supermasks in Superposition (Wortsman et al., 2020).”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “We compare against representative class-incremental learning methods, including DualNet [43], SupSup [61], DynaER [64] and RPSNet [48].”
Class-Incremental Learning with Strong Pre-trained Models (2022, background): “The exceptions infer the task identity typically by learning a routing classifier [1, 4], or estimating confidence scores [27].”
Memory Efficient Continual Learning with Transformers (2022, methodology): “Recently, some studies use pre-trained models for class incremental learning [26, 21]; (4) Parameter isolation-based methods [25, 62] allocate model parameters dedicated to different tasks and mask them out when learning a new task; (5) Meta-learning-based methods, which directly optimize the…”
Architecture Matters in Continual Learning (2022, methodology): “Perhaps the most related side of algorithms to our work is the parameter-isolation methods where diﬀerent parts of the model are devoted to a speciﬁc task (Yoon et al., 2018; Mallya and Lazebnik, 2018; Wortsman et al., 2020).”
Learning to Prompt for Continual Learning (2021, methodology): “SupSup [60] and DualNet [44] are both based on ResNet18, recommended by their original authors.”
Kernel Continual Learning (2021, background): “This is achieved by a gating mechanism (Wortsman et al., 2020; Masse et al., 2018), or by incrementally adding new parameters to the model (Rusu et al., 2016).”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, methodology): “In our experiments, we compare MARK with recent state of the art methods: Hard Attention to the Tasks (HAT) [7], A-GEM [28], Adversarial Continual Learning (ACL) [18], GPM [29], Experience Replay and SupSup[30].”
Posterior Meta-Replay for Continual Learning (2021, background): “Follow-up work also achieves task inference via predictive uncertainty, e.g., Wortsman et al. [78] use it to select a learned binary mask per task that modulates a random base network.”
Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “Inspired by Wortsman et al. (2020), we propose to utilize the perplexity of each adapter over the input X as a measure of uncertainty.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “A related approach achieves a similar goal by leveraging the concept of superposition, where context information is stored for each task, so that the weights can be explicitly decomposed into orthogonal sub-networks (Cheung et al., 2019; Wortsman et al., 2020).”
Efficient Continual Learning with Modular Networks and Task-Driven Priors (2020, methodology): “Most recently, Wortsman et al. (2020) have proposed a stream with 2500 tasks all derived from MNIST permutations.”