Boosting Adversarial Training with Hypersphere Embedding

By Tianyu Pang, Xiao Yang, Yinpeng Dong, Kun Xu, Hang Su, Jun Zhu. 2020

170 citations recorded by the source corpus.

Adversarial training (AT) is one of the most effective defenses against adversarial attacks for deep learning models. In this work, we advocate incorporating the hypersphere embedding (HE) mechanism into the AT procedure by regularizing the features onto compact manifolds, which constitutes a lightweight yet effective module to blend in the strength of representation learning. Our extensive analyses reveal that AT and HE are well coupled to benefit the robustness of the adversarially trained models from several aspects. We validate the effectiveness and adaptability of HE by embedding it into the popular AT frameworks including PGD-AT, ALP, and TRADES, as well as the FreeAT and FastAT strategies. In the experiments, we evaluate our methods under a wide range of adversarial attacks on the CIFAR-10 and ImageNet datasets, which verifies that integrating HE can consistently enhance the model robustness for each AT framework with little extra computation.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Alan L. Yuille
3. Jianfeng Gao
4. Jun Zhu
5. Mohit Bansal
6. Bernt Schiele
7. Quoc V. Le
8. Jingjing Liu

In Safety, Robustness & Interpretability

1. Matthias Hein
2. Tianyu Pang
3. Sven Gowal
4. Jun Zhu
5. Yinpeng Dong
6. Timothy A. Mann
7. Hang Su
8. Xiao Yang

Citations in the corpus

• Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Among existing defense approaches, adversarial training (AT) is a general strategy to empower models with state-of-the-art robustness in different settings [65, 42, 72, 83, 50].”
• Adversarial Examples Improve Image Recognition (2019, background): “Since the first discovery of the vulnerability of ConvNets to adversarial attacks [40], many efforts [2, 7, 15, 16, 18, 23, 29, 36, 42, 45, 50] have been made to improve network robustness.”
• RobustBench: a standardized adversarial robustness benchmark (2020, background): “[92] (and 21% on average) 283 compared to 4% for the standard model.”
• Advances in adversarial attacks and defenses in computer vision: A survey (2021, methodology): “Another related method [363] proposes to boost adversarial training by embedding a hypersphere method in the training process by regularizing features onto a compact manifolds.”
• Data Augmentation Can Improve Robustness (2021, background): “Indeed, since Madry et al. [33], various modifications to their original implementation have been proposed [20, 26, 39, 43, 56, 62].”
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples (2020, background): “Since Madry et al. (2017), various modifications to their algorithm have been proposed (Xie et al., 2018; Pang et al., 2020b; Huang et al., 2020; Qin et al., 2019; Zoran et al., 2019; Andriushchenko & Flammarion, 2020).”
• Relating Adversarially Robust Generalization to Flat Minima (2021, unknown): “Originally proposed in different variants in [56, 40, 24], it received considerable attention in [37, 16] and has been extended in various ways: [33, 7, 2] utilize interpolated or unlabeled examples, [57, 38] achieve robustness against multiple threat models, [54, 32, 64] augment AT with a reject op”
• Bag of Tricks for Adversarial Training (2020, background): “…from other domains, including ensemble learning (Tramèr et al., 2018; Pang et al., 2019), metric learning (Mao et al., 2019; Li et al., 2019; Pang et al., 2020c), generative modeling (Jiang et al., 2018; Pang et al., 2018b; Wang & Yu, 2019; Deng et al., 2020), semi-supervised learning…”
• Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, methodology): “ADV-COMP-SUM with the smallest model WRN-28-10 surpasses (Pang et al., 2020a; Rice et al., 2020; Qin et al., 2019).”
• Fixing Data Augmentation to Improve Adversarial Robustness (2021, background): “(2018), various modification to their original implementation have been proposed (Zhang et al., 2019; Xie et al., 2019; Pang et al., 2020; Huang et al., 2020; Qin et al., 2019; Rice et al., 2020; Gowal et al., 2020).”
• Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “…data, several enhancements for AT have been made employing strategies inspired by other areas, including metric learning (Mao et al., 2019; Pang et al., 2020a;b), self-supervised learning (Chen et al., 2020a;b; Naseer et al., 2020; Wang & Liu, 2022), ensemble learning (Tramèr et al.,…”
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “Recently, adversarial training (AT) [69, 74, 55, 50, 79, 71] has shown success on standard V+L tasks [16, 65].”