Learning multiple visual domains with residual adapters

By Sylvestre-Alvise Rebuffi, Hakan Bilen, Andrea Vedaldi. 2017

1,069 citations recorded by the source corpus.

There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to recognize well uniformly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “For each of the following datasets, we use the version collated, resized, and split into train/validation/test sets by Rebuffi et al. (2017).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “Residual Adapters were first introduced for adapting vision models in Rebuffi et al. (2017), but their formulation used a single projection layer,”
Parameter-Efficient Transfer Learning for NLP (2019, methodology): “Instead, we present an alternative transfer method based on adapter modules (Rebuffi et al., 2017).”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “More generally, work on task prompts is closely aligned with work on “adapters” (Rebuffi et al., 2017; Houlsby et al., 2019), small bottleneck layers inserted between frozen pre-trained network layers.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, background): “For example, Zhang et al. (2020a) trains a “side” network that is fused with the pretrained model via summation; adapter-tuning inserts task-specific layers (adapters) between each layer of the pretrained LM (Houlsby et al., 2019; Lin et al., 2020; Rebuffi et al., 2017; Pfeiffer et al., 2020).”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Other supervised learning benchmarks focus on performance on multiple data sets, either via transfer learning, meta-learning, or multi-task learning [49, 61].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Vision Benchmarks The Visual Decathlon (Rebufﬁ et al., 2017) contains ten classiﬁcation tasks: Omniglot, and nine using natural images.”
LoRA: Low-Rank Adaptation of Large Language Models (2021, methodology): “[15, 34] propose inserting adapter layers between existing layers in a neural network.”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “Recently, adapters (Rebufﬁ et al., 2017; Houlsby et al., 2019) have emerged as a solution that overcomes the problems of catastrophic forgetting and imbalanced training set sizes.”
Training Neural Networks with Fixed Sparse Masks (2021, background): “For example, Adapters [19, 41, 5] introduce additional parameters into a pre-trained model in the form of small task-speciﬁc modules that are ﬁne-tuned while the rest of the model’s parameters are kept ﬁxed.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, background): “Among these approaches, adapters, which belong to the ﬁrst category, are widely used in computer vision [41, 42] and natural language processing [16, 20, 33].”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “The meta-learner either learns to produce new parameters directly from the new data [9, 33, 56, 62, 64, 72, 73], or learns to produce an update rule to iteratively optimize the base learner to fit the new data [2, 6, 8, 38, 63, 97].”
Visual Prompt Tuning (2022, background): “Transfer learning has been extensively studied for vision tasks in the context of ConvNets [86] and many techniques have been introduced including side tuning [81], residual adapter [63], bias tuning [6], etc.”