The Pitfalls of Simplicity Bias in Neural Networks

By Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, Praneeth Netrapalli. 2020

478 citations recorded by the source corpus.

Several works have proposed Simplicity Bias (SB)---the tendency of standard training procedures such as Stochastic Gradient Descent (SGD) to find simple models---to justify why neural networks generalize well [Arpit et al. 2017, Nakkiran et al. 2019, Valle-Perez et al. 2019]. However, the precise notion of simplicity remains vague. Furthermore, previous settings that use SB to justify why neural networks generalize well do not simultaneously capture the brittleness of neural networks---a widely observed phenomenon in practice [Goodfellow et al. 2014, Jo and Bengio 2017]. To this end, we introduce a collection of piecewise-linear and image-based datasets that (a) naturally incorporate a precise notion of simplicity and (b) capture the subtleties of neural networks trained on real datasets. Through theory and experiments on these datasets, we show that SB of SGD and variants is extreme: neural networks rely exclusively on the simplest feature and remain invariant to all predictive complex features. Consequently, the extreme nature of SB explains why seemingly benign distribution shifts and small adversarial perturbations significantly degrade model performance. Moreover, contrary to conventional wisdom, SB can also hurt generalization on the same data distribution, as SB persists even when the simplest feature has less predictive power than the more complex features. We also demonstrate that common approaches for improving generalization and robustness---ensembles and adversarial training---do not mitigate SB and its shortcomings. Given the central role played by SB in generalization and robustness, we hope that the datasets and methods in this paper serve as an effective testbed to evaluate novel algorithmic approaches aimed at avoiding the pitfalls of extreme SB.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Competency Problems: On Finding and Removing Artifacts in Language Data (2021, background): “Here, we ask whether typical NLP models learn to bias their predictions based on these artifacts for both the SNLI and RTE (Dagan et al., 2005) 9 datasets.”
Domain Generalization using Causal Matching (2020, methodology): “The features vary in their simplicity, a measure of the simplicity of the feature is given by the number of linear pieces in the optimal classiﬁ-cation/decision curve (Figure 1, (Shah et al., 2020)).”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “[87] showed that neural networks tend to rely entirely on the simple features.”
Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition (2019, background): “Although this approach is promising, it alas comes with the substantial caveat that purely discriminative networks are prone to encode noise as features [13] or fall for a most simple discriminative solution that neglects meaningful features [14].”
What shapes feature representations? Exploring datasets, architectures, and training (2020, background): “non-linear) ones are more reliable [37], and Parascandolo et al.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “Recent work aims at identifying the reasons behind this phenomenon [97, 70, 22, 73, 51, 76, 100, 92, 83, 105, 42, 79, 4], while complementary work quantifies resulting shortcomings, including poor generalization to out-of-distribution (OOD) test data, reliance upon spurious correlations, and lack of”