Improving the Generalization of Adversarial Training with Domain Adaptation
By Chuanbiao Song, Kun He, Liwei Wang, John E. Hopcroft. 2018
148 citations recorded by the source corpus.
By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models. However, most existing adversarial training approaches are based on a specific type of adversarial attack. It may not provide sufficiently representative samples from the adversarial domain, leading to a weak generalization ability on adversarial examples from other attacks. Moreover, during the adversarial training, adversarial perturbations on inputs are usually crafted by fast single-step adversaries so as to scale to large datasets. This work is mainly focused on the adversarial training yet efficient FGSM adversary. In this scenario, it is difficult to train a model with great generalization due to the lack of representative adversarial samples, aka the samples are unable to accurately reflect the adversarial domain. To alleviate this problem, we propose a novel Adversarial Training with Domain Adaptation (ATDA) method. Our intuition is to regard the adversarial training on FGSM adversary as a domain adaption task with limited number of target domain samples. The main idea is to learn a representation that is semantically meaningful and domain invariant on the clean domain as well as the adversarial domain. Empirical evaluations on Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA can greatly improve the generalization of adversarial training and the smoothness of the learned models, and outperforms state-of-the-art methods on standard benchmark datasets. To show the transfer ability of our method, we also extend ATDA to the adversarial training on iterative attacks such as PGD-Adversial Training (PAT) and the defense performance is improved considerably.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “When us-ing DA, we modify the robust training algorithm by adding the regularization loss proposed by Song et al. [51].”
- Metric Learning for Adversarial Robustness (2019, methodology): “Following the visualization technique in [28, 30, 33], we investigate what happens to the latent representations as they undergo attack.”
- Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “Moreover, poor generalization of adversarial training is also often highlighted in the literature [328]–[331].”
- Robust Learning from Untrusted Sources (2019, background): “…a vast body of literature focusing on robustness of learning algorithms to corruptions within a dataset, e.g. (Tukey, 1960; Huber, 2011; Diakonikolas et al., 2016; Prasad et al., 2018), and on identifying data corrup-tions at prediction time , e.g. (Hendrycks & Gimpel, 2017; Sun & Lampert, 2018).”
- The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization (2020, background): “To circumvent these issues, practitioners introduce adversarial training in order to produce adversarially robust models [15, 34, 23, 43, 12, 35] that can still make consistently correct predictions, even when faced with perturbed data.”
- Boosting Adversarial Training with Hypersphere Embedding (2020, background): “This causes the training process to devote noneffective efforts to increasing ‖z‖ for easy examples and consequently overlook the hard ones, which leads to vicious circles and could degrade the model robustness against strong adversarial attacks [51, 59].”
- Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks (2019, background): “…of models by adversarial training (Goodfellow et al., 2014), learning how to generate adversarial examples with high transferability is important and has gained increasing attentions in the literature (Liu et al., 2016; Dong et al., 2018; Xie et al., 2019; Dong et al., 2019; Wang et al., 2019).”
- Towards Deep Learning Models Resistant to Large Perturbations (2020, background): “…been numerous studies to improve the adversarial accuracy of adversarial training on the test set by using techniques such as domain adaptation (Song et al., 2018) or label smoothing (Shafahi et al., 2018), while some others have tried to decrease the computational cost of adversarial training…”
- High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks (2019, background): “, 2018), yet shall be aware that newer works are introduced proliferatively (e.g., Ilyas et al., 2018; Yan et al., 2018; Guo et al., 2018b; Alaifari et al., 2019; Xiao et al., 2019; Song et al., 2019; Farnia et al., 2019; Wong et al., 2019).”
- Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, methodology): “…et al., 2020), weight averaging/perturbation (Gowal et al., 2020; Wu et al., 2020; Tsai et al., 2021; Yu et al., 2018; Prabhu et al., 2019) and other techniques (Zhang & Wang, 2019; Qin et al., 2019; Goldblum et al., 2020; Song et al., 2019; Pang et al., 2020b; Lee et al., 2020; Qian et al., 2021).”