Interpreting Adversarially Trained Convolutional Neural Networks

By Tianyuan Zhang, Zhanxing Zhu. 2019

172 citations recorded by the source corpus.

We attempt to interpret how adversarially trained convolutional neural networks (AT-CNNs) recognize objects. We design systematic approaches to interpret AT-CNNs in both qualitative and quantitative ways and compare them with normally trained models. Surprisingly, we find that adversarial training alleviates the texture bias of standard CNNs when trained on object recognition tasks, and helps CNNs learn a more shape-biased representation. We validate our hypothesis from two aspects. First, we compare the salience maps of AT-CNNs and standard CNNs on clean images and images under different transformations. The comparison could visually show that the prediction of the two types of CNNs is sensitive to dramatically different types of features. Second, to achieve quantitative verification, we construct additional test datasets that destroy either textures or shapes, such as style-transferred version of clean data, saturated images and patch-shuffled ones, and then evaluate the classification accuracy of AT-CNNs and normal CNNs on these datasets. Our findings shed some light on why AT-CNNs are more robust than those normally trained ones and contribute to a better understanding of adversarial training over CNNs from an interpretation perspective.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Jun Zhu
3. Quoc V. Le
4. Bernt Schiele
5. Hang Su
6. Nicholas Carlini
7. Tianyu Pang
8. Matthias Hein

Citations in the corpus

• Adversarial Examples Improve Image Recognition (2019, background): “Though such trained models have several nice properties as described in [51, 47, 43], they cannot generalize well to clean images [23, 45].”
• Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “Moreover, poor generalization of adversarial training is also often highlighted in the literature [329], [330], [331], [332].”
• Do Adversarially Robust ImageNet Models Transfer Better? (2020, background): “However, a recent line of work has studied adversarially robust models in their own right, casting (1) as a prior on learned feature representations [Eng+19a; Ily+19; Jac+19; ZZ19].”
• Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2020, result): “A similar finding was found in other image datasets (Zhang and Zhu, 2019).”
• Boosting Adversarial Training with Hypersphere Embedding (2020, background): “Furthermore, previous studies observe that the adversarial examples against robust models exhibit salient data characteristics (Tsipras et al., 2019; Tao et al., 2018; Zhang & Zhu, 2019; Ilyas et al., 2019).”
• Permuted AdaIN: Reducing the Bias Towards Global Statistics in Image Classification (2020, background): “Zhang and Zhu [47] show that adversarial training reduces texture bias.”
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization (2020, background): “There is a large body of work dedicated to adversarially robust models [Zhang and Zhu, 2019, Santurkar et al., 2019, Zhang et al., 2019, Bhagoji et al., 2019, Diochnos et al., 2019, Wei and Ma, 2019].”
• Partial success in closing the gap between human and machine vision (2021, background): “…silver lining: when testing whether models are biased towards texture or shape by testing them on cue conflict images (Figure 3), in accordance with [69, 70] we observe a perfect relationship between shape bias and the degree of adversarial training, a big step in the direction of human shape bias…”
• Testing Robustness Against Unforeseen Adversaries (2019, background): “Going one step further, we believe that robustness itself provides only a narrow window into a model’s behavior; in addition to robustness, we should seek to build a diverse toolbox for understanding machine learning models, including visualization [36, 37], disentanglement of relevant features [27]”