Superposition of many models into one

By Brian Cheung, Alexander V. Terekhov, Yubei Chen, Pulkit Agrawal, Barbara Olshausen. 2019

127 citations recorded by the source corpus.

We present a method for storing multiple models within a single set of parameters. Models can coexist in superposition and still be retrieved individually. In experiments with neural networks, we show that a surprisingly large number of models can be effectively stored within a single parameter instance. Furthermore, each of these models can undergo thousands of training steps without significantly interfering with other models within the superposition. This approach may be viewed as the online complement of compression: rather than reducing the size of a network after training, we make use of the unrealized capacity of a network during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Once for All: Train One Network and Specialize it for Efficient Deployment (2019, neutral): “, Wo) (Cheung et al., 2019), we only require 7.”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, methodology): “(3) Parameter isolation methods augment the networks (Rusu et al., 2016; Lee et al., 2017a; Aljundi et al., 2017) or decompose the network into subnetworks for each task (Mallya & Lazebnik, 2018; Cheung et al., 2019; Yoon et al., 2020).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “A related approach achieves a similar goal by leveraging the concept of superposition, where context information is stored for each task, so that the weights can be explicitly decomposed into orthogonal sub-networks (Cheung et al., 2019; Wortsman et al., 2020).”
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, methodology): “…on developing methods for mitigating forgetting, with techniques to implicitly or explicitly partition parameters for diﬀerent tasks [14, 25, 26, 27], regularization (elasticity) and Bayesian approaches [6, 15, 28, 29, 18, 30] as well as directly storing data from earlier tasks in…”
Supermasks in Superposition (2020, methodology): “Baselines & Ablations Figure 5 (left) shows that even in Scenario GNu, SupSup is able to outperform PSP [4] and BatchE [51] in Scenario GG—methods using task identity.”
Continual Learning via Neural Pruning (2019, methodology): “However, in comparison to [1] where the superposition is approximate, our approach is exact and we have precise control over the relationship between the different models being superposed in terms of transfer learning.”
Unsupervised Model Personalization While Preserving Privacy and Scalability: An Open Problem (2020, background): “[3] superpose different models into a single one from which task-specific parameters can be retrieved.”
Building a Subspace of Policies for Scalable Continual Learning (2022, methodology): “Cheung et al., 2019; Wortsman et al., 2020).”
Gradient Regularized Contrastive Learning for Continual Domain Adaptation (2020, background): “Several other papers (Wulfmeier, Bewley, and Posner 2018; Hoffman et al. 2014; Wang, He, and Katabi 2020; Cheung et al. 2019) presented continuous domain adaptation with the emphasis to generalize on a transitioning target domain.”
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks (2019, methodology): “lidation (EWC). A constraint-based substitutive approach from [12]. We use the formulation from [30] which scales better. Parameter Superposition (PSP): A parameter-masking substitutive approach from [4] that attempts to make tasks independent from one another by mapping the weights to approximately”
Deep Ensembles for Low-Data Transfer Learning (2020, methodology): “nsembles end-to-end with diversity-encouraging losses, such as those in Lee et al. (2015) and Webb et al. (2019). Lastly, works such as Batch Ensembles (Wen et al., 2020) and Parameter Superposition (Cheung et al., 2019) systematically decompose network parameters to compactly train ensembles. For p”
OvA-INN: Continual Learning with Invertible Neural Networks (2020, background): “Recent works in networks merging could alleviate the memory issue by sharing weights [32] or relying on weights superposition [33].”
How fine can fine-tuning be? Learning efficient language models (2020, methodology): “Also, note that randomly generated high-dimensional masks can also support multi-task learning, e.g. Cheung et al. (2019).”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, background): “Although some recent works explore the combination of different approaches [2, 29, 54] or new strategies [74, 57, 14], our benchmark aims to provide an in-depth analysis of these three basic lifelong learning directions to reveal their pros and cons on robot learning tasks.”