Robust fine-tuning of zero-shot models

By Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt. 2021

1,012 citations recorded by the source corpus.

Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution. On ImageNet and five derived distribution shifts, WiSE-FT improves accuracy under distribution shift by 4 to 6 percentage points (pp) over prior work while increasing ImageNet accuracy by 1.6 pp. WiSE-FT achieves similarly large robustness gains (2 to 23 pp) on a diverse set of six further distribution shifts, and accuracy gains of 0.8 to 3.3 pp compared to standard fine-tuning on commonly used transfer learning datasets. These improvements come at no additional computational cost during fine-tuning or inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “These capabilities can be improved for a given target distribution through finetuning, but at the cost of generality and robustness to distribution shifts (Wortsman et al. 2021).”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “…another, ﬁne-tuning a model on downstream tasks can sometimes reduce out-of-distribution performance (Radford et al., 2021; Andreassen et al., 2021; Wortsman et al., 2021; Pham et al., 2021), and the best single model on the target distribution may not be the best model on out-of-distribution…”
Conditional Prompt Learning for Vision-Language Models (2022, background): “With the rise of such powerful vision-language models, the community has recently started to investigate potential solutions to efficiently adapt these models to downstream datasets [14, 53, 56, 62].”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “Furthermore, CoOp [70], CLIP-Adapter [17] and WiSE-FT [64] indicate that the performance of CLIP can be significantly improved with either abundant or limited training data by fine-tuning weight-”
Meta-learning via Language Model In-context Tuning (2021, background): “Can we combine in-context learning and gradient learning to get the beneﬁt of both worlds (Wortsman et al., 2021)?”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, background): “On top of that, CoOp [74], CLIP-Adapter [16] and WiSE-FT [61] significantly improve the CLIP with limited training data by freezing the pre-trained weights and training additive learnable modules.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, positive): “Parameter Averaging Our averaging mechanism is inspired by the discovery that averaging many ﬁne-tuned vision models improves out-of-domain generalization (Wortsman et al., 2022a; Izmailov et al., 2018).”
Sigmoid Loss for Language Image Pre-Training (2023, background): “Follow-up works show that contrastively pre-trained models produce good representations for fine-tuning [39, 12], linear regression [18], object detection [25], semantic segmentation [26] and video tasks [43].”
Git Re-Basin: Merging Models modulo Permutation Symmetries (2022, background): “We therefore anticipate its potential application in ﬁelds such as ﬁnetuning (Devlin et al., 2019; Wortsman et al., 2022b;a), federated learning (McMa-han et al., 2017; Kone ˇ cn ´ y et al., 2016a;b), and model patching (Matena & Raffel, 2021; Sung et al., 2021; Raffel, 2021).”
Scaling Open-Vocabulary Object Detection (2023, methodology): “A second important limitation of our method, similar to other open-vocabulary models [24, 22, 31], is the trade-off between fine-tuned and open-vocabulary performance addressed in Section 4.”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, neutral): “We considered three strategies to mitigate reward overﬁtting: • LoRA Scaling : multiplying the LoRA weights by a constant < 1 (see Section 5.2), which interpolates between the pre-trained model and ﬁne-tuned model, similarly to Wortsman et al. (2022).”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, background): “The resulting models have also shown unprecedented robustness to data distribution shifts when compared to standard models trained on a target distribution [53, 71].”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, methodology): “…recognition, allowing visual recognition models linear BEiT [1] ViT-B 224 2 83.2 37.6 47.1 MAE [17] ViT-B 224 2 83.6 68.0 48.1 SimMIM [45] ViT-B 224 2 83.8 56.7 47.6 SimMIM [45] Swin-B 224 2 84.8 24.8 48.3 WiSE-FT CLIP [40] ViT-L 336 2 87.1 - - DINO [3] ViT-B to classify almost any category.”