Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
By Raphael Gontijo Lopes, Sylvia J. Smullin, Ekin Dogus Cubuk, Ethan Dyer. 2020
87 citations recorded by the source corpus.
Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood. In practice, augmentation policies are often chosen using heuristics of either distribution shift or augmentation diversity. Inspired by these, we seek to quantify how data augmentation improves model generalization. To this end, we introduce interpretable and easy-to-compute measures: Affinity and Diversity. We find that augmentation performance is predicted not by either of these alone but by jointly optimizing the two.
Subfields: Artificial Intelligence, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Weizhu Chen
- Jiawei Han
- Mohit Bansal
- Eduard H. Hovy
- Yelong Shen
- Varun Prashant Gangal
- Soroush Vosoughi
In Optimization & Training
- Jason Wei
- Sarath Chandar
- Adam Prügel-Bennett
- Mert Bulent Sariyildiz
- Eduard H. Hovy
- Jiawei Han
- Mahesan Niranjan
- Steven Y. Feng
Citations in the corpus
- A Survey of Data Augmentation Approaches for NLP (2021, background): “A recent work in vision (Gontijo-Lopes et al., 2020) has proposed that affinity (the distributional shift caused by DA) and diversity (the complexity of the augmentation) can predict DA performance, but it is unclear how these results might translate to NLP. Minimal benefit for pretrained models on…”
- Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics (2020, background): “[13] Raphael Gontijo-Lopes, Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer.”
- FMix: Enhancing Mixed Sample Data Augmentation (2020, background): “Gontijo-Lopes et al. (2020) proposes two measures to explain the impact of augmentation on generalisation when jointly optimised: affinity and diversity.”
- CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, background): “In addition, it has been shown that the diversity of augmented samples plays a vital role in their effectiveness (Xie et al., 2019; Gontijo-Lopes et al., 2020).”
- Hard Negative Mixing for Contrastive Learning (2020, background): “A number of recent studies [10, 17, 39] show that carefully handcrafting the set of data augmentations applied to images is instrumental in learning such represen-”