Improving Robustness using Generated Data

By Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan A. Calian, Timothy A. Mann. Neural Information Processing Systems, 2021

379 citations recorded by the source corpus.

Recent work argues that robust training requires substantially larger datasets than those required for standard classification. On CIFAR-10 and CIFAR-100, this translates into a sizable robust-accuracy gap between models trained solely on data from the original training set and those trained with additional data extracted from the"80 Million Tiny Images"dataset (TI-80M). In this paper, we explore how generative models trained solely on the original training set can be leveraged to artificially increase the size of the original training set and improve adversarial robustness to $\ell_p$ norm-bounded perturbations. We identify the sufficient conditions under which incorporating additional generated data can improve robustness, and demonstrate that it is possible to significantly reduce the robust-accuracy gap to models trained with additional real data. Surprisingly, we even show that even the addition of non-realistic random data (generated by Gaussian sampling) can improve robustness. We evaluate our approach on CIFAR-10, CIFAR-100, SVHN and TinyImageNet against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large absolute improvements in robust accuracy compared to previous state-of-the-art methods. Against $\ell_\infty$ norm-bounded perturbations of size $\epsilon = 8/255$, our models achieve 66.10% and 33.49% robust accuracy on CIFAR-10 and CIFAR-100, respectively (improving upon the state-of-the-art by +8.96% and +3.29%). Against $\ell_2$ norm-bounded perturbations of size $\epsilon = 128/255$, our model achieves 78.31% on CIFAR-10 (+3.81%). These results beat most prior works that use external data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9c8d46b59e871e18d8d2e1ec1aa9b96d2f3d7342

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Unlike methods that use model-based rendering, here we focus on the use of data-driven generative models of natural images, for which GANs have remained the predominant approach to date [6, 17, 36].”
Diffusion Models for Adversarial Purification (2022, methodology): “…WideResNet-70-16 (Gowaletal.,2020) 91.10 66.02 (Rebufﬁ et al., 2021) 92.23 68.56 (Gowal et al., 2020) 85.29 59.57 (Rebufﬁ et al., 2021) 88.54 64.46 (Gowal et al., 2021) 88 (Gowaletal.,2020) 94.74 79.88 (Rebufﬁ et al., 2021) 95.74 81.44 (Gowal et al., 2020) 90.90 74.03 (Rebufﬁ et al., 2021) 92…”
Fixing Data Augmentation to Improve Adversarial Robustness (2021, background): “Since its original publication (2 Mar 2021), this paper has been accepted to NeurIPS 2021 as two separate and updated papers (Rebuffi et al., 2021; Gowal et al., 2021).”
Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “While Gowal et al. (2021) report that CutMix is compatible with using generated data, preliminary experiments in Pang et al. (2022) suggest that the effectiveness of CutMix may be dependent on the specific implementation.”