Adversarial Logit Pairing

By Harini Kannan, Alexey Kurakin, Ian J. Goodfellow. 2018

666 citations recorded by the source corpus.

In this paper, we develop improved techniques for defending against adversarial examples at scale. First, we implement the state of the art version of adversarial training at unprecedented scale on ImageNet and investigate whether it remains effective in this setting - an important open scientific question (Athalye et al., 2018). Next, we introduce enhanced defenses using a technique we call logit pairing, a method that encourages logits for pairs of examples to be similar. When applied to clean examples and their adversarial counterparts, logit pairing improves accuracy on adversarial examples over vanilla adversarial training; we also find that logit pairing on clean examples only is competitive with adversarial training in terms of accuracy on two datasets. Finally, we show that adversarial logit pairing achieves the state of the art defense on ImageNet against PGD white box attacks, with an accuracy improvement from 1.5% to 27.9%. Adversarial logit pairing also successfully damages the current state of the art defense against black box attacks on ImageNet (Tramer et al., 2018), dropping its accuracy from 66.6% to 47.1%. With this new accuracy drop, adversarial logit pairing ties with Tramer et al.(2018) for the state of the art on black box attacks on ImageNet.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Local Relation Networks for Image Recognition (2019, methodology): “We utilize the hyperparameters from [16] of the attacking methods, and employ the targeted multi-step PGD adversarial method for training with the same hyper-parameters except for the number of attack steps, set to 16 due to limited GPU resources.”
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “ALP is an adversarial example defense for large-scale image classifiers (Kannan et al., 2018).”
Counterfactual Fairness in Text Classification through Robustness (2018, methodology): “bset of adversarial examples. The robustness literature has attempted to address adversarial examples using a variety of techniques, such as adversarial training [4, 16] and adversarial logit pairing [8]. Several papers draw connections between fairness, text generation, and robustness. Landeiro and”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, background): “The Jensen-Shannon Consistency Loss impels to model to be stable, consistent, and insensitive across a diverse range of inputs (Bachman et al., 2014; Zheng et al., 2016; Kannan et al., 2018).”
Adversarial Training for Free! (2019, methodology): “Table 3 shows the robustness of Resnet-50 on ImageNet with (cid:15) = 2 .”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “A number of defenses have been proposed [43], [44], [45], [46], [47], yet many have shown to be less effective against an adaptive adversary [48], [49], [50], [51].”
Feature Denoising for Improving Adversarial Robustness (2018, methodology): “We first compare with ALP [10], the previous state-of-the-art.”
Certified Adversarial Robustness via Randomized Smoothing (2019, background): “, 2019), and have suggested that Gaussian data augmentation could supplement or replace adversarial training (Zantedeschi et al., 2017; Kannan et al., 2018).”
Using Videos to Evaluate Image Model Robustness (2019, background): “The distribution of distances between consecutive frames have a mean of 213 and standard deviation of 49.1, which is much larger than what is typically considered for adversarial examples (less than an (cid:15) = 16 in L ∞ space (Kannan et al. (2018)).”
TSViz: Demystification of Deep Learning Models for Time-Series Analysis (2018, background): “well-studied in prior literature, specifically for image-based classification networks [27].”
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, methodology): “Instead, we choose to adversarially pre-train a Downsampled ImageNet model against an untargeted adversary, contra Kurakin et al. (2017); Kannan et al. (2018); Xie et al. (2018).”
Motivating the Rules of the Game for Adversarial Example Research (2018, methodology): “The evaluation in Kannan, Kurakin, and Goodfellow [5] shows one way to select an appropriate model for a variety of possible test sets (Figure 5).”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, methodology): “We also compare TRADES with Adversarial Logit Pairing (ALP) (Kannan et al., 2018; Engstrom et al., 2018).”
Adversarial Examples Improve Image Recognition (2019, methodology): “Meanwhile, recent works [18, 16, 45] also suggest that training with adversarial examples on large datasets, e.”
Unlabeled Data Improves Adversarial Robustness (2019, methodology): “For the supervised robust loss, we use a robustness-promoting regularization term proposed in [56] and closely related to earlier proposals in [57, 30, 20].”