Scalable Transfer Learning with Expert Models

By Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, Neil Houlsby. 2020

72 citations recorded by the source corpus.

Transfer of pre-trained representations can improve sample efficiency and reduce computational requirements for new tasks. However, representations used for transfer are usually generic, and are not tailored to a particular distribution of downstream tasks. We explore the use of expert representations for transfer with a simple, yet effective, strategy. We train a diverse set of experts by exploiting existing label structures, and use cheap-to-compute performance proxies to select the relevant expert for each target task. This strategy scales the process of transferring to new tasks, since it does not revisit the pre-training data during transfer. Accordingly, it requires little extra compute per target task, and results in a speed-up of 2-3 orders of magnitude compared to competing approaches. Further, we provide an adapter-based architecture able to compress many experts into a single model. We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, background): “And while most work has been in natural language processing, they have also been successfully used in a variety of domains including computer vision (Puigcerver et al., 2020), speech recognition (You et al., 2021) and multi-modal learning (Mustafa et al., 2022).”
Mixture-of-Experts with Expert Choice Routing (2022, background): “Conditional Computation: Computation decisions can be made dynamically based on the input [23, 25].”
Memory Efficient Continual Learning with Transformers (2022, background): “There are a number of works that explores task transferability in NLP [42, 32, 59, 45, 44]”
ImageNet-21K Pretraining for the Masses (2021, methodology): “[34] and [37] used ImageNet-21K and JFT-300M to train expert models according to the datasets hierarchies, and com-”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, methodology): “Recently, [43] proposed first training on the base dataset and then training with subsets of the base dataset to create specialist models, and finally using the target data to select the best specialist model.”
Deep Ensembles for Low-Data Transfer Learning (2020, methodology): “• We extend the work from Puigcerver et al. (2020) and demonstrate the efficacy of kNN accuracy as a cheap proxy metric for selecting a subset of candidate pre-trained models.”
Learning a Universal Template for Few-shot Dataset Generalization (2021, background): “A representative approach that is related to FLUTE is that of Puigcerver et al. (2021): they ﬁrst train a set of diverse ‘experts’ on different data and then, for each downstream task, they select the most appropriate expert and ﬁne-tune it on that task.”
Memory Efficient Continual Learning for Neural Text Classification (2022, background): “There are a number of works that explores task transferability in NLP (Phang et al., 2018; Liu et al., 2019; Vu et al., 2020; Puigcerver et al., 2020; Pruksachatkun et al., 2020).”