The Implicit Bias of Gradient Descent on Separable Data

By Daniel Soudry, Elad Hoffer, Suriya Gunasekar, Nati Srebro. 2017

1,145 citations recorded by the source corpus.

We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The result also generalizes to other monotone decreasing loss functions with an infimum at infinity, to multi-class problems, and to training a weight layer in a deep network in a certain restricted setting. Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself. This can help explain the benefit of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases. Our methodology can also aid in understanding implicit regularization n more complex models and with other optimization methods.

Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Percy Liang
  3. Sergey Levine
  4. Quoc V. Le
  5. Ruslan Salakhutdinov
  6. Mohit Bansal
  7. Jianfeng Gao
  8. Dacheng Tao

In Optimization & Training

  1. Yuanzhi Li
  2. Jason D. Lee
  3. Tengyu Ma
  4. Zeyuan Allen-Zhu
  5. Simon Lacoste-Julien
  6. Sebastian U. Stich
  7. Simon Shaolei Du
  8. Martin Jaggi

In Scaling & Learning Theory

  1. Yuanzhi Li
  2. Tengyu Ma
  3. Jason D. Lee
  4. Quanquan Gu
  5. Simon Shaolei Du
  6. Samet Oymak
  7. Colin Wei
  8. Yoshua Bengio

Citations in the corpus