Domain Adaptive Transfer Learning with Specialist Models

By Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V. Le, Ruoming Pang. 2018

116 citations recorded by the source corpus.

Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance. We find that more pre-training data does not always help, and transfer performance depends on a judicious choice of pre-training data. These findings are important given the continued increase in dataset sizes. We further propose domain adaptive transfer learning, a simple and effective pre-training method using importance weights computed based on the target dataset. Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning. Our methods achieve state-of-the-art results on multiple fine-grained classification datasets and are well-suited for use in practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “(2) Compared to state-of-the-art models, including DAT (Ngiam et al., 2018) that dynamically synthesizes training data and GPipe (Huang et al., 2018) that is trained with specialized pipeline parallelism, our EfﬁcientNet models still surpass their accuracy in 5 out of 8 datasets, but using 9.6x fewe”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “eNet-21k dataset [10], a public dataset containing 14.2 million images and 21k classes organized by the WordNet hierarchy. Images may contain multiple labels. BiT-L is trained on the JFT-300M dataset [51,38,61]. This dataset is a newer version of that used in [18,8]. JFT-300M consists of around 300 ”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “We used the version from [47], which filtered the validation set of ImageNet.”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, neutral): “, 2018), DAT (Ngiam et al., 2018), BiT-M/L (Kolesnikov et al.”
Federated Visual Classification with Real-World Data Distribution (2020, methodology): “Importance sampling is a classic technique for variance reduction in Monte Carlo methods [12,9] and has been used widely in domain adaption literature for countering covariate and target shift [23,34,21].”
What is being transferred in transfer learning? (2020, background): “4 Related work Recent work has looked into whether transfer learning is always successful [18, 25, 31, 20, 12].”
ImageNet-21K Pretraining for the Masses (2021, background): “This kind of 129 incomplete tagging methodology, which is common in large datasets [31, 41], hinders and complicates 130 the training process.”
Scalable Transfer Learning with Expert Models (2020, background): “The second suite is a collection of popular natural datasets commonly used in transfer learning literature and, particularly in (Ngiam et al., 2018), to which we compare our work.”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, methodology): “Similarly, [37] used target data to reweight the importance of base data.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “[39] found more data for pre-training does not always help and proposed to use importance weights to select pre-training data.”
In-domain representation learning for remote sensing (2019, background): “Mahajan et al. (2018) explored training bigger models on larger datasets with weaker labels and (Ngiam et al., 2018; Cui et al., 2018) showed gains by closer matching the domains.”
Measuring Dataset Granularity (2019, background): “This has been also verified by recent work on domain-specific adaptive transfer learning [16, 6, 39].”
Deep Ensembles for Low-Data Transfer Learning (2020, methodology): “, 2009), but recent works show considerable improvements by careful, task-specific pre-trained model selection (Ngiam et al., 2018; Puigcerver et al., 2020).”
Large Scale Learning of General Visual Representations for Transfer (2019, methodology): “BiT-L is trained on the JFT-300M dataset, as in [40,33,50].”
Transfusion: Understanding Transfer Learning for Medical Imaging (2019, background): “Despite the immense popularity of transfer learning in medical imaging, there has been little work studying its precise effects, even as recent work on transfer learning in the natural image setting [11, 16, 20, 12, 7] has challenged many commonly held beliefs.”
Transfusion: Understanding Transfer Learning with Applications to Medical Imaging (2019, unknown): “Despite the popularity of transfer learning, its exact bene ts are poorly understood, with recent papers (He et al., 2018; Ngiam et al., 2018; Kornblith et al., 2018) challenging deep-rooted prior assumptions.”
Rethinking the Hyperparameters for Fine-tuning (2020, unknown): “Ngiam et al. (2018) ﬁnd that when domains are mismatched, the effectiveness of transfer learning is negative, even when domains are intuitively similar.”
A Survey of Deep Learning for Scientific Discovery (2020, background): “Note however that the precise eﬀects of transfer learning are not yet fully understood, and an active research area [116, 184, 266, 159, 143, 181, 235] looks at investigating its exact properties.”