Intriguing properties of neural networks
By Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, Rob Fergus. 2013
16,794 citations recorded by the source corpus.
Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Percy Liang
- Quoc V. Le
- Sergey Levine
- Jianfeng Gao
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Nicolas Papernot
- Bo Li
- Dawn Song
- Florian Tramèr
- Aleksander Madry
- Cho-Jui Hsieh
Citations in the corpus
- Generative Adversarial Nets (2014, neutral): “In practice, equation”
- What makes ImageNet good for transfer learning? (2016, background): “An interesting observation that CNNs have “blindspots” was made by [43].”
- Adversarial Training Methods for Semi-Supervised Text Classification (2016, background): “Traditional adversarial and virtual adversarial training can be interpreted both as a regularization strategy [28, 6, 22] and as defense against an adversary who can supply malicious inputs [28, 6].”
- Shortcut learning in deep neural networks (2020, background): “tainty is incredibly big. To humans only some of these look like guitars, others like patterns (interpretable or abstract) and many more resemble white noise or even look like airplanes, cats or food [8,45,41]. Figure4on the right, for example, highlights a variety of image pairs that have hardly an”
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Finally, a large body of work aims to learn adversarial examples that cause downstream models to produce spurious outputs [50], some of which are reviewed in [59].”
- Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “Neural networks are susceptible to adversarial examples—changes to the input which do or do not change the network’s prediction in surprising ways (Szegedy et al., 2014; Jia and Liang, 2017; Ribeiro et al., 2018; Alzantot et al., 2018).”
- Robust Neural Machine Translation with Joint Textual and Phonetic Embedding (2018, background): “…are complicated, and some of them are: 1) neural networks are inherently sensitive to noises, such as adversarial examples (Goodfellow et al., 2014; Szegedy et al., 2013), 2) the global effect of attention, where every input word can affect every output word generated by the decoder, and 3) the…”
- Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “[68, 24, 55, 22], have shown that computer vision models lack robustness.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Common examples include noise corruptions [33, 38], spatial transformations [28, 29], and adversarial examples [5, 84].”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Adversarial Training Adversarial machine learning is an active research area [63, 16, 5].”
- Natural Adversarial Examples (2019, background): “[19] define adversarial examples [49] as “inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake.”
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Most work on robustness in deep learning methods for vision has focused on the important challenges of robustness to adversarial examples (Szegedy et al., 2014; Carlini & Wagner, 2017; 2016), unknown unknowns (Hendrycks et al.”
- Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “[50] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J.”
- Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “, finding adversarial examples misclassified as a target class [46].”
- ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “Recent work in the context of deep learning has investigated sequence patterns (Joulin and Mikolov, 2015), combinatorial problems (Vinyals et al., 2015), or executing programming language code (Zaremba and Sutskever, 2014), amongst others.”
- Explaining and Harnessing Adversarial Examples (2014, background): “Szegedy et al. (2014b) demonstrated a variety of intriguing properties of neural networks and related models.”
- Towards Robust Vision Transformer (2021, background): “Early works focus on the model safety under the adversarial examples with constrained perturbations [11, 38].”
- Randaugment: Practical automated data augmentation with a reduced search space (2019, background): “dation accuracy [7, 49], robustness [40, 46, 9], or both [25].”
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, background): “Many researches have shown that deep models are easily fooled by small and unrecognizable perturbations on the input images, a phenomenon referred to as adversarial attacks [9, 38].”