The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

By Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, Rebecca Roelofs. 2021

94 citations recorded by the source corpus.

Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models. Models that are more accurate on the out-of-distribution data relative to this baseline exhibit"effective robustness"and are exceedingly rare. Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “For another, ﬁne-tuning a model on downstream tasks can sometimes reduce out-of-distribution performance (Radford et al., 2021; Andreassen et al., 2021; Wortsman et al., 2021; Pham et al., 2021), and the best single model on the target distribution may not be the best model on out-of-distribution…”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, background): “3) Standard fine-tuning of CLIP models can significantly reduce their robustness to data distribution shifts and generality [53, 2, 71, 70]—key strengths of CLIP we want to leverage.”
Scaling Vision Transformers to 22 Billion Parameters (2023, background): “While fine-tuning boosts accuracy on both ImageNet and out-of-distribution datasets, the effective robustness (Andreassen et al., 2021) decreases (Figure 5).”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “C V ] 2 1 Fe b 20 Zhai et al., 2020; He et al., 2020),1 and even when testing OOD, prior work usually fine-tunes all parameters of their model (Hendrycks et al., 2019a; Miller et al., 2021; Andreassen et al., 2021).”
No One Representation to Rule Them All: Overlapping Features of Training Methods (2021, background): “They report impressive results on out-of-distribution benchmarks, and have been shown to have higher dominance probabilities (Andreassen et al., 2021).”
Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry (2025, unknown): “[7 , 8, 9, 10, 11] These observations have motivated research on model merging and weight-averaging techniques that aim to reconcile the gains of fine-tuning with the robustness of pre-trained checkpoints.”