Deep Ensembles for Low-Data Transfer Learning

By Basil Mustafa, Carlos Riquelme, Joan Puigcerver, andAndr'e Susano Pinto, Daniel Keysers, Neil Houlsby. 2020

27 citations recorded by the source corpus.

In the low-data regime, it is difficult to train good supervised models from scratch. Instead practitioners turn to pre-trained models, leveraging transfer learning. Ensembling is an empirically and theoretically appealing way to construct powerful predictive models, but the predominant approach of training multiple deep networks with different random initialisations collides with the need for transfer via pre-trained weights. In this work, we study different ways of creating ensembles from pre-trained models. We show that the nature of pre-training itself is a performant source of diversity, and propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset. The approach is simple: Use nearest-neighbour accuracy to rank pre-trained models, fine-tune the best ones with a small hyperparameter sweep, and greedily construct an ensemble to minimise validation cross-entropy. When evaluated together with strong baselines on 19 different downstream tasks (the Visual Task Adaptation Benchmark), this achieves state-of-the-art performance at a much lower inference budget, even when selecting from over 2,000 pre-trained models. We also assess our ensembles on ImageNet variants and show improved robustness to distribution shift.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ali Farhadi
2. Hannaneh Hajishirzi
3. Ludwig Schmidt
4. Mohit Bansal
5. Percy Liang
6. Luke S. Zettlemoyer
7. Quoc V. Le
8. Gabriel Ilharco

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ali Farhadi
3. Ming-Hsuan Yang
4. Trevor Darrell
5. Dacheng Tao
6. Xiaogang Wang
7. Andrea Vedaldi
8. Jitendra Malik

In Radiology, Nuclear Medicine and Imaging

1. Ludwig Schmidt
2. Mitchell Wortsman
3. Gabriel Ilharco
4. Asaf Noy
5. Tal Ridnik
6. Emanuel Ben-Baruch
7. Ali Farhadi
8. Hongseok Namkoong

In Representation & Self-Supervised Learning

1. Ali Farhadi
2. Ludwig Schmidt
3. Mitchell Wortsman
4. Gabriel Ilharco
5. Simon Kornblith
6. Jong Wook Kim
7. Asaf Noy
8. Tal Ridnik

In Meta & Continual Learning

1. Ali Farhadi
2. Gabriel Ilharco
3. Mitchell Wortsman

Citations in the corpus

• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “[66] propose a method for identifying subsets of pre-trained models for fine-tuning and later ensembling them, finding strong in-distribution accuracy and robustness to distribution shift.”
• ImageNet-21K Pretraining for the Masses (2021, methodology): “[34] and [37] used ImageNet-21K and JFT-300M to train expert models according to the datasets hierarchies, and com-”
• Robust fine-tuning of zero-shot models (2021, background): “[71] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby.”