Bad Global Minima Exist and SGD Can Reach Them

By Shengchao Liu, Dimitris Papailiopoulos, Dimitris Achlioptas. 2019

89 citations recorded by the source corpus.

Several recent works have aimed to explain why severely overparameterized models, generalize well when trained by Stochastic Gradient Descent (SGD). The emergent consensus explanation has two parts: the first is that there are "no bad local minima", while the second is that SGD performs implicit regularization by having a bias towards low complexity models. We revisit both of these ideas in the context of image classification with common deep neural network architectures. Our first finding is that there exist bad global minima, i.e., models that fit the training set perfectly, yet have poor generalization. Our second finding is that given only unlabeled training data, we can easily construct initializations that will cause SGD to quickly converge to such bad global minima. For example, on CIFAR, CINIC10, and (Restricted) ImageNet, this can be achieved by starting SGD at a model derived by fitting random labels on the training data: while subsequent SGD training (with the correct labels) will reach zero training error, the resulting model will exhibit a test accuracy degradation of up to 40% compared to training from a random initialization. Finally, we show that regularization seems to provide SGD with an escape route: once heuristics such as data augmentation are used, starting from a complex model (adversarial initialization) has no effect on the test accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Group-Theoretic Framework for Data Augmentation (2019, background): “Other works have demonstrated connections between data augmentation and kernel classifiers [25], marginalized corrupted features [52], network architecture [9], margins [56], certain complexity measures [39], optimization landscape [48], and biological plausibility [36].”
SGD on Neural Networks Learns Functions of Increasing Complexity (2019, methodology): “[27] also finds an initialization for SGD that leads to poor generalization, using a slightly different technique.”