Spectrally-normalized margin bounds for neural networks

By Peter L. Bartlett, Dylan J. Foster, Matus Telgarsky. 2017

1,473 citations recorded by the source corpus.

This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity": their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for a standard AlexNet network trained with SGD on the mnist and cifar10 datasets, with both original and random labels; the bound, the Lipschitz constants, and the excess risks are all in direct correlation, suggesting both that SGD selects predictors whose complexity scales with the difficulty of the learning task, and secondly that the presented bound is sensitive to this complexity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Sergey Levine
  3. Percy Liang
  4. Quoc V. Le
  5. Ruslan Salakhutdinov
  6. Mohit Bansal
  7. Jianfeng Gao
  8. Dacheng Tao

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Scaling & Learning Theory

  1. Tengyu Ma
  2. Yuanzhi Li
  3. Jason D. Lee
  4. Samet Oymak
  5. Colin Wei
  6. Quanquan Gu
  7. Greg Yang
  8. Behnam Neyshabur

Citations in the corpus