MMA Training: Direct Input Space Margin Maximization through Adversarial Training

By Gavin Weiguang Ding, Yash Sharma, Kry Yik-Chau Lui, Ruitong Huang. 2018

312 citations recorded by the source corpus.

We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the ``shortest successful perturbation'', demonstrating a close connection between adversarial losses and the margins. We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. Instead of adversarial training with a fixed $\eps$, MMA offers an improvement by enabling adaptive selection of the ``correct'' $\eps$ as the margin individually for each datapoint. In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins. Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Example Games (2020, background): “Robust optimization techniques aim to improve the robustness of a classiﬁer by learning model parameters by incorporating adversarial examples from a given attack into the training process [51, 64, 20].”
advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch (2019, methodology): “Robust Training: Adversarially augmented training (Kurakin et al., 2016; Madry et al., 2017; Ding et al., 2018) and provably robust training (Wong and Kolter, 2018; Mirman et al., 2018; Gowal et al., 2018) have been shown to be the most eﬀective methods against worst-case perturbations.”
Cross-Entropy Loss Functions: Theoretical Analysis and Applications (2023, background): “, 2023b), curriculum and adaptive attack methods (Cai et al., 2018; Wang et al., 2019; Zhang et al., 2020a; Ding et al., 2022; Cheng et al., 2022), efficient adversarial training (Shafahi et al.”
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks (2020, unknown): “54 14 (Ding et al., 2020) 84.”
Geometry-aware Instance-reweighted Adversarial Training (2020, methodology): “, 2018), MMA (Ding et al., 2020), and DAT (Wang et al.”
Smooth Adversarial Training (2020, background): “Existing works suggest that, to further adversarial robustness, we need to either sacriﬁce accuracy on clean inputs [68, 69, 79, 13], or incur additional computational cost [38, 75, 74].”
Recent Advances in Adversarial Training for Adversarial Robustness (2021, methodology): “28% [Ding et al., 2020] Wide ResNet PGD100 8/255 CIFAR-10 47.”
Relating Adversarially Robust Generalization to Flat Minima (2021, unknown): “Originally proposed in different variants in [56, 40, 24], it received considerable attention in [37, 16] and has been extended in various ways: [33, 7, 2] utilize interpolated or unlabeled examples, [57, 38] achieve robustness against multiple threat models, [54, 32, 64] augment AT with a reject op”
Adversarial Robustness with Semi-Infinite Constrained Learning (2021, methodology): “In practice, penalty-based methods that incorporate clean data into the training objective are often used to overcome this issue [53–56].”
Adversarial Robustness through Local Linearization (2019, methodology): “Others, such as work by Ding et al [7] adaptively increase the perturbation radius by ﬁnd the minimal length perturbation which changes the output label.”
Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, unknown): “, 2014), and numerous defense methods have been invented (Madry et al., 2018; Sinha et al., 2018; Rony et al., 2019; Zhang et al., 2019a; Ding et al., 2020).”
RobustBench: a standardized adversarial robustness benchmark (2020, background): “[32] aim at input-adaptive robustness as opposed to robustness within a single 233 `p-radius.”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, methodology): “A margin-maximization variant of adversarial training was proposed in [335] that creates adversarial examples using sample-specific η (see Eq.”