No One Representation to Rule Them All: Overlapping Features of Training Methods

By Raphael Gontijo Lopes, Yann Dauphin, Ekin Dogus Cubuk. 2021

72 citations recorded by the source corpus.

Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy ~76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ali Farhadi
2. Hannaneh Hajishirzi
3. Ludwig Schmidt
4. Mohit Bansal
5. Percy Liang
6. Luke S. Zettlemoyer
7. Quoc V. Le
8. Gabriel Ilharco

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ali Farhadi
3. Trevor Darrell
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Andrea Vedaldi
7. Jitendra Malik
8. Xiaogang Wang

In Representation & Self-Supervised Learning

1. Ali Farhadi
2. Ludwig Schmidt
3. Mitchell Wortsman
4. Gabriel Ilharco
5. Simon Kornblith
6. Jong Wook Kim
7. Ari S. Morcos
8. S. Gadre

In Statistical Learning Theory

1. Ali Farhadi
2. Ludwig Schmidt
3. Gabriel Ilharco
4. Hongseok Namkoong
5. Rebecca Roelofs
6. Simon Kornblith
7. Ari S. Morcos
8. Mitchell Wortsman

Citations in the corpus

• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “For the ensemble baselines (Dietterich, 2000; Lakshmi-narayanan et al., 2017) we ensemble the logits (unormalized outputs) of models as in Gontijo-Lopes et al. (2022).”
• Robust fine-tuning of zero-shot models (2021, background): “[30] Raphael Gontijo-Lopes, Yann Dauphin, and Ekin D Cubuk.”
• Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “(8)We speculate that the random ensemble is poor because its constituent models make correlated errors during evaluation (Gontijo-Lopes et al., 2022).”