Does Data Augmentation Lead to Positive Margin?

By Shashank Rajput, Zhili Feng, Zachary B. Charles, Po-Ling Loh, Dimitris Papailiopoulos. 2019

38 citations recorded by the source corpus.

Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness. DA artificially expands the training set by applying random noise, rotations, crops, or even adversarial perturbations to the input data. Although DA is widely used, its capacity to provably improve robustness is not fully understood. In this work, we analyze the robustness that DA begets by quantifying the margin that DA enforces on empirical risk minimizers. We first focus on linear separators, and then a class of nonlinear models whose labeling is constant within small convex hulls of data points. We present lower bounds on the number of augmented data points required for non-zero margin, and show that commonly used DA techniques may only introduce significant margin after adding exponentially many points to the data set.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, background): “Rajput et al. (2019) show that DA can increase the positive margin for classifiers, but only when augmenting exponentially many examples for common DA methods.”
UniformAugment: A Search-free Probabilistic Data Augmentation Approach (2020, background): “Rajput et al. [13] analyzed how addictive spherical data augmentations aﬀects the margin of classiﬁers.”
A Group-Theoretic Framework for Data Augmentation (2019, background): “Other works have demonstrated connections between data augmentation and kernel classiﬁers [25], marginalized corrupted features [52], network architecture [9], margins [56], certain complexity measures [39], optimization landscape [48], and biological plausibility [36].”
On the Generalization Effects of Linear Transformations in Data Augmentation (2020, background): “Previous work has analyzed the effect of Gaussian augmentation (Rajput et al., 2019) and feature averaging effect of adding transformed data (Dao et al., 2019; Chen et al., 2019).”
Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, background): “analyzed the performance of data augmentation through the lens of margin [31].”