Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

By Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson. 2022

483 citations recorded by the source corpus.

Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Jeff Wu
3. Pavel Izmailov
4. Jan Leike
5. Leo Gao
6. Luke S. Zettlemoyer
7. Sergey Levine
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Pavel Izmailov
2. Jeff Wu
3. Jan Leike
4. Bowen Baker
5. Leo Gao
6. OpenAI
7. Collin Burns
8. Yining Chen

Citations in the corpus

• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Aside from image classification, a few works have focused on other computer vision problems, such as image semantic segmentation [106,107], object detection [108], and analogical reasoning [109].”
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “…Utama et al., 2020; Creager et al., 2021; Idrissi et al., 2022; Orgad & Belinkov, 2023), training classifiers with more favorable inductive biases (Kirichenko et al., 2023; Zhang et al., 2022; Iskander et al., 2024), or editing out undesired concepts (Iskander et al., 2023; Belrose et al., 2023;…”
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “Recent approaches to mitigating this problem include weight ensembling (Wortsman et al., 2022b;a), finetuning only a subset of layers (Kirichenko et al., 2023; Lee et al., 2022a), or mitigating the distortion effects that finetuning has on pre-trained features (Kumar et al., 2022).”