Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
By Anh Totti Nguyen, Jason Yosinski, Jeff Clune. 2014
3,543 citations recorded by the source corpus.
Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification problems. Given that DNNs are now able to classify objects in images with near-human-level performance, questions naturally arise as to what differences remain between computer and human vision. A recent study [30] revealed that changing an image (e.g. of a lion) in a way imperceptible to humans can cause a DNN to label the image as something else entirely (e.g. mislabeling a lion a library). Here we show a related result: it is easy to produce images that are completely unrecognizable to humans, but that state-of-the-art DNNs believe to be recognizable objects with 99.99% confidence (e.g. labeling with certainty that white noise static is a lion). Specifically, we take convolutional neural networks trained to perform well on either the ImageNet or MNIST datasets and then find images with evolutionary algorithms or gradient ascent that DNNs label with high confidence as belonging to each dataset class. It is possible to produce images totally unrecognizable to human eyes that DNNs believe with near certainty are familiar objects, which we call “fooling images” (more generally, fooling examples). Our results shed light on interesting differences between human vision and current DNNs, and raise questions about the generality of DNN computer vision.
Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Bernt Schiele
- Andrea Vedaldi
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Nicolas Papernot
- Bo Li
- Aleksander Madry
- Dawn Song
- Patrick Drew McDaniel
- Cho-Jui Hsieh
Citations in the corpus
- Shortcut learning in deep neural networks (2020, background): “generalisation (Figure 4) [45, 36, 46, 41].”
- The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, background): “• Functional approximation error – this is a secondary type of error, which stems from our function approx-imators being insufficiently expressive (or sometimes too expressive outside of the original distribution support [Nguyen et al., 2015]).”
- Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “However, machine learning (ML) models frequently assign wrong labels with high confidence to OoD examples, such as adversarial examples [46, 29]—inputs specially crafted by an adversary to cause a target model to misbehave.”
- ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “Investigations for image recognition (Szegedy et al., 2014; Nguyen et al., 2015; Zhang et al., 2017) have shown surprising behavior very different from what would be expected of their “surpassing human-level performance” (He et al.”
- Explaining and Harnessing Adversarial Examples (2014, methodology): “These experiments suggest that the optimization algorithms employed by Nguyen et al. (2014) are overkill (or perhaps only needed on ImageNet), and that the rich geometric structure in their fooling images are due to the priors encoded in their search procedures, rather than those structures being…”
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop (2015, background): “As a result, deep networks trained on them often learn features that are noisy and/or unstable [17, 11].”
- CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, background): “A key issue in synthesis through optimization is that the produced images often leave the space of natural images (Nguyen et al., 2015), or fool the system through adversarial means (Goodfellow et al., 2014b), thus a body of work aims to discover ‘natural image priors’ to constrain which images may…”
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, background): “The approach described in the previous section is able to generate text tuned for a particular discriminator, but left unchecked it will quickly result in unrealistic adversarial or fooling examples (Szegedy et al., 2013; Nguyen et al., 2015) as the text moves into low probability regions.”
- Towards the Science of Security and Privacy in Machine Learning (2016, background): “It is therefore not surprising that randomly sampled inputs can be constrained to be classified in a class with confidence [52], [67].”
- Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, methodology): “At a high level, some penalizes the image to remove high-frequency noise [66, 70, 72, 79, 98]; some searches for images that can still We used the implementations from https://github.”
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, methodology): “Similar optimization techniques can also be used to generate high-confidence fooling images [18,19].”
- Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “Computer vision presents a particularly striking challenge: very small changes to the input image can fool state-of-the-art neural networks with high probability [23, 7, 15, 21, 14].”
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, neutral): “This is consistent with a great deal of anecdotal evidence from researchers (Nguyen & O’Connor, 2015; Yu et al., 2010; Provost et al., 1998; Nguyen et al., 2015).”
- Synthesizing the preferred inputs for neurons in neural networks via deep generator networks (2016, background): “Previous studies have shown that doing so without biasing the images produced creates unrealistic, uninterpretable images [5, 3], because the set of all possible images is so vast that it is possible to produce ‘fooling’ images that excite a neuron, but do not resemble the natural images that neuron”
- Understanding Neural Networks Through Deep Visualization (2015, methodology): “Instead, they are composed of a collection of “hacks” that happen to cause high (or low) activations: extreme pixel values, structured high frequency patterns, and copies of common motifs without global structure (Simonyan et al., 2013; Nguyen et al., 2014; Szegedy et al., 2013; Goodfellow et al., 2”