A Kernel Theory of Modern Data Augmentation

By Tri Dao, Albert Gu, Alexander J. Ratner, Virginia Smith, Christopher De Sa, Christopher R'e. 2018

225 citations recorded by the source corpus.

Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines. In this paper, we seek to establish a theoretical framework for understanding data augmentation. We approach this from two directions: First, we provide a general model of augmentation as a Markov process, and show that kernels appear naturally with respect to this model, even when we do not employ kernel classification. Next, we analyze more directly the effect of augmentation on kernel classifiers, showing that data augmentation can be approximated by first-order feature averaging and second-order variance regularization components. These frameworks both serve to illustrate the ways in which data augmentation affects the downstream learning model, and the resulting analyses provide novel connections between prior work in invariant kernels, tangent propagation, and robust optimization. Finally, we provide several proof-of-concept applications showing that our theory can be useful for accelerating machine learning workflows, such as reducing the amount of computation needed to train using augmented data, and predicting the utility of a transformation prior to training.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, background): “Dao et al. (2019) think of DA transformations as kernels, and ﬁnd two ways DA helps: averaging of features and variance regularization.”
Zero-Shot Knowledge Distillation in Deep Networks (2019, methodology): “which has proven useful in further boosting the model performance (Dao et al., 2018).”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “Recent work in this area includes learning better transformations [9, 52, 53], inferring combinations of transformations [6], unsupervised methods [49], theory of data augmentation [7], and applications for one-shot learning [1].”
Does Data Augmentation Lead to Positive Margin? (2019, methodology): “Dao et al. (2018) study data augmentation as feature-averaging and variance regularization, using a Markov process to augment the dataset.”
Neural Kernels Without Tangents (2020, background): “Finding new paths to cheaply augment kernels (Ratner et al., 2017; Dao et al., 2019) or to incorporate the symmetries implicit in data augmentation explicitly in kernels should dramatically improve the effectiveness of kernel methods on contemporary datasets.”
A Fourier-based Framework for Domain Generalization (2021, methodology): “Similar as in [3, 13], we expand R̂aug with Taylor expansion:”
Data Augmentation for Spoken Language Understanding via Joint Variational Generation (2018, result): “Data Augmentation and Regularization For data-hungry models, appropriate regularization is necessary to achieve high performance for many tasks (Dao et al. 2018).”
A Group-Theoretic Framework for Data Augmentation (2019, background): “For example, DeVries and Taylor (2017b) proposes to randomly mask out square regions of input images and fill the regions with pure gray color; Zhong et al.”
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss (2021, methodology): “Similar Markov chains on augmentated data have also been used in previous work (Dao et al., 2019) to study properties of data augmentation.”
On the Generalization Effects of Linear Transformations in Data Augmentation (2020, background): “Dao et al. (2019) propose a kernel theory to show that label-invariant augmentations are equivalent to transforming the kernel matrix in a way that incorporates the prior of the transformation.”
Leveraging molecular structure and bioactivity with chemical language models for de novo drug design (2023, methodology): “Future improvement might be possible by combining nucleus sampling with data augmentation for CLM transfer learning 33,64 – 66 .”
Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style (2021, background): “While this intuitively makes sense and has shown remarkable empirical results, the success of data augmentation techniques in practice is still not very well understood from a theoretical perspective—despite some efforts [17, 19, 28].”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, background): “The few related works are of DA in the image domain, considering data augmentation as encoding a priori knowledge about data or task invariance [140], variance reduction [141] or regularization methods [142].”
A Comprehensive Survey of Image Augmentation Techniques for Deep Learning (2022, methodology): “Recently, kernel theory [102] and group theory [103] have been used to analyze the eﬀects of image augmentation.”
On Data Augmentation for GAN Training (2020, background): “fake) to specific transformations, and penalizing model complexity via a regularization term based on the variance of the augmented forms [50].”
Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, methodology): “As the analysis in [9], we expand each term of the objective function with data augmentation using Taylor expansion:”
Tradeoffs in Data Augmentation: An Empirical Study (2021, methodology): “With this deﬁnition, it is widely acknowledged that commonly-used augmentations act as regular-izers (Hernández-García & König, 2018a;b; Zhang et al., 2016; Dao et al., 2019).”