In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness

By Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, Percy Liang. 2020

68 citations recorded by the source corpus.

Consider a prediction setting where a few inputs (e.g., satellite images) are expensively annotated with the prediction targets (e.g., crop types), and many inputs are cheaply annotated with auxiliary information (e.g., climate information). How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, we show that (i) using auxiliary information as input features improves in-distribution error but can hurt out-of-distribution (OOD) error; while (ii) using auxiliary information as outputs of auxiliary tasks to pre-train a model improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, neutral): “Self-training (Lee, 2013; McClosky et al., 2006; Kumar et al., 2020; Xie et al., 2021) applies a trained model to unlabeled data, obtains predicted targets (pseudolabels), and uses them as extra training examples.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “…world includes not only poverty mapping but also tracking child mortality (Osgood-Zimmerman et al., 2018; Reiner et al., 2018; Burke et al., 2016), educational attainment (Graetz et al., 2018), food security and crop yield prediction (Wang et al., 2020; You et al., 2017; Xie et al., 2020).”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “In particular, ID and OOD accuracy are not correlated, unlike Recht et al. (2018) but like Xie et al. (2021a).”
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift (2022, methodology): “In some datasets, such as satellite remote sensing datasets [Jean et al., 2016, Xie et al., 2021], a standard model can latch onto country-speciﬁc features that may be less prevalent OOD.”