Efficient Parametrization of Multi-domain Deep Neural Networks

By Sylvestre-Alvise Rebuffi, Hakan Bilen, Andrea Vedaldi. 2018

387 citations recorded by the source corpus.

A practical limitation of deep neural networks is their high degree of specialization to a single task and visual domain. Recently, inspired by the successes of transfer learning, several authors have proposed to learn instead universal feature extractors that, used as the first stage of any deep network, work well for several tasks and domains simultaneously. Nevertheless, such universal features are still somewhat inferior to specialized networks. To overcome this limitation, in this paper we propose to consider instead universal parametric families of neural networks, which still contain specialized problem-specific models, but differing only by a small number of parameters. We study different designs for such parametrizations, including series and parallel residual adapters, joint adapter compression, and parameter allocations, and empirically identify the ones that yield the highest compression. We show that, in order to maximize performance, it is necessary to adapt both shallow and deep layers of a deep network, but the required changes are very small. We also show that these universal parametrization are very effective for transfer learning, where they outperform traditional fine-tuning techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, methodology): “In vision, convolutional adapter modules have been studied (Rebuffi et al., 2017; 2018; Rosenfeld & Tsotsos, 2018).”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, methodology): “At the other end of the spectrum, adapter methods [1, 15] that insert trainable transformations at different layers of the pretrained model require more parameters than the aforementioned approaches but are more memory-efficient and obtain performance comparable to full fine-tuning [1, 16].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, background): “Among these approaches, adapters, which belong to the first category, are widely used in computer vision [41, 42] and natural language processing [16, 20, 33].”
Visual Prompt Tuning (2022, neutral): “Prior research has also looked at adding additional residual blocks (or adapters) to the backbone [64,81].”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “In domain adaptation [20, 43], residual adapter [18, 25, 26] is proposed to learn style information related to new domains, thus improving the overall generalization performance of the network.”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “There are three main approaches to mitigating catastrophic forgetting [53]: 1) increasing model capacity to include new representations that will not interfere with old knowledge [67, 60, 58, 3, 57, 54, 46, 45, 65], 2) regularizing continual parameter updates such that parameters do not deviate too ”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “97 Parallel RA [50] 70.”
IL2M: Class Incremental Learning With Dual Memory (2019, background): “The authors of [23] introduce universal parametric families of neural networks which share a majority of parameters and use small modular adapters that are attached to the network to specialize it for new tasks.”
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning (2019, background): “new task solving network in terms of a linear combination of the parameters of a fixed base network [24, 23].”
Don't forget, there is more than forgetting: new metrics for Continual Learning (2018, methodology): “(Rebuffi et al., 2018) dataset is used: each task consists of a training batch of 10 (disjoint) classes at a time.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, methodology): “In particular, Adapters (Rebuffi et al., 2018), which learn to read and write to layers of a shared model, have been applied to obtain parameter-efficient BERT models (Houlsby et al.”
MultiGrain: a unified image embedding for classes and instances (2019, background): “Ongoing lines of work include finding the right architecture for an efficient sharing of parameters [36], and finding the right optimization parameters for such networks in order to depart from the traditional setting of single-task single-dataset end-to-end gradient descent, and efficiently weight ”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, background): “C om pl ex ity The model evolves by adding parameters and weights to interconnect them [42, 48, 56] or small networks [2] to include new knowledge.”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “As an alternative to fine-tuning all the model’s parameters, prior work (Houlsby et al., 2019; Rebuffi et al., 2018; Stickland and Murray, 2019) inserted small modules called adapter layers within layers of a pretrained model, as shown in Figure 1.”
Scalable Transfer Learning with Expert Models (2020, background): “Instead of relying on fixed universal representations, [50, 51, 52] use small additional modules, or adapters, that incorporate knowledge from several visual domains.”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “We show the results of SpotTune and the baselines on the Visual Decathlon Challenge in Table 3.”
Adding New Tasks to a Single Network with Weight Trasformations using Binary Masks (2018, methodology): “entry of the 2017 edition of the challenge [34] (DAN), the network with taskspecific residual [31] (RA) and parallel [32] (PA) adapters.”
Learning More Universal Representations for Transfer-Learning (2017, methodology): “[6] (by adding data labeled among specific categories) and [18], [19] (by adding data labeled among generic categories); splitting variations on the set of categories has been performed in [15], [24],”