Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

By Mitchell Wortsman, Gabriel Ilharco, S. Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith and 1 more. 2022

1,632 citations recorded by the source corpus.

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results"model soups."When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[127] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael GontijoLopes, Ari S.”
Stable and low-precision training for large-scale vision-language models (2023, methodology): “We examine CLIP-style models because of their importance in computer vision: CLIP-style models reach state-ofthe-art performance on a wide range of image classification tasks [46, 65, 44, 7] and underlie image generation methods such as DALLE·2 [49] and Stable Diffusion [51].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “, integrating multiple minima in parameter or function space along the low-error path, since connecting them ensures flatness on that path [59], [129], [182], [308], [430], [450].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “The only additional technique added is model souping (Wortsman et al., 2022): we run the 900K to 1M cool-down twice, once with inception cropping and once with resizing only.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, positive): “…effective way to develop domain-speciﬁc language models (Gururangan et al., 2020), and parameter interpolation techniques work best with models that have a shared initialization (Izmailov et al., 2018; Frankle et al., 2020; Wortsman et al., 2022b; Matena and Raffel, 2021; Wortsman et al., 2022a) .”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, background): “Work on this problem has led to significant improvements along several threads including image level predictions like image classification [13, 54, 58], object level predictions like object detection [33,51,53,59–61], as well as pixel level predictions like semantic segmentation [10,29,51,53].”
Token Merging: Your ViT But Faster (2022, background): “We sweep over ViT-B/16 MAE fine-tuning on ImageNet-1k with r = 16 and find that the default parameters from the official code release work the best. layer decay epochs acc 0.875 50 82.83 0.825 50 83.52 0.750 50 83.57 0.750 75 83.35 0.750 100 82.33 (b) Video Fine-Tuning.”
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, background): “We average checkpoints towards the end of training [21]; refer to Appendix A.”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “We also explored changing the random seed during SFT, and then using those models to create model soups [Wortsman et al., 2022].”
Merging Models with Fisher-Weighted Averaging (2021, methodology): “2 These checkpoints were fine-tuned with a variety of hyperparameter settings that were not chosen by us, so our experimental setting most closely matches the “Model Soup” approach [67].”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, neutral): “For example, only fine-tuning on relevant slicing technique (Zhang et al., 2022), model soup (Wortsman et al., 2022a), fine-tuning then interpolating (Wortsman et al., 2022b), etc.”
The Stable Signature: Rooting Watermarks in Latent Diffusion Models (2023, background): “Model providers could deploy their models to different user groups with a unique watermark, and monitor that they are used in a responsible manner.”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, neutral): “For model merging algorithms, we compare the performance of RegMean with the previously introduced methods of simple averaging ( Simple ) (Wortsman et al., 2022) and Fisher-weighted averaging ( Fisher ) (Matena & Raffel, 2021).”
FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling (2023, methodology): “…performance by amplifying differences between models in the ensemble ranging from varying the data augmentation used for each model (Stickland & Murray, 2020), the architecture (Zaidi et al., 2021), the hyperparameters (Wortsman et al., 2022), and even the training objectives (Jain et al., 2020).”