MaxUp: A Simple Way to Improve Generalization of Neural Network Training

By Chengyue Gong, Tongzheng Ren, Mao Ye, Qiang Liu. 2020

59 citations recorded by the source corpus.

We propose \emph{MaxUp}, an embarrassingly simple, highly effective technique for improving the generalization performance of machine learning models, especially deep neural networks. The idea is to generate a set of augmented data with some random perturbations or transforms and minimize the maximum, or worst case loss over the augmented data. By doing so, we implicitly introduce a smoothness or robustness regularization against the random perturbations, and hence improve the generation performance. For example, in the case of Gaussian perturbation, \emph{MaxUp} is asymptotically equivalent to using the gradient norm of the loss as a penalty to encourage smoothness. We test \emph{MaxUp} on a range of tasks, including image classification, language modeling, and adversarial certification, on which \emph{MaxUp} consistently outperforms the existing best baseline methods, without introducing substantial computational overhead. In particular, we improve ImageNet classification from the state-of-the-art top-1 accuracy $85.5\%$ without extra data to $85.8\%$. Code will be released soon.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Sergey Levine
  3. Yoshua Bengio
  4. Trevor Darrell
  5. Percy Liang
  6. Mohit Bansal
  7. Pieter Abbeel
  8. Ruslan Salakhutdinov

In Optimization & Training

  1. Soham De
  2. Samuel L. Smith
  3. Andrew Brock
  4. Karen Simonyan
  5. Dawn Song
  6. Avi Schwarzschild
  7. Micah Goldblum
  8. Xinyun Chen

Citations in the corpus