A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions

By Samuel F. Dodge, Lina Karam. 2017

464 citations recorded by the source corpus.

Deep neural networks (DNNs) achieve excellent performance on standard classification tasks. However, under image quality distortions such as blur and noise, classification accuracy becomes poor. In this work, we compare the performance of DNNs with human subjects on distorted images. We show that, although DNNs perform better than or on par with humans on good quality images, DNN performance is still much lower than human performance on distorted images. We additionally find that there is little correlation in errors between DNNs and human subjects. This could be an indication that the internal representation of images are different between DNNs and the human visual system. These comparisons with human performance could be used to guide future development of more robust DNNs.

Subfields: Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Industrial and Manufacturing Engineering

In Media Technology

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “However, research in the subsequent years has repeatedly found that these models still make many simple mistakes (Dodge & Karam, 2017; Geirhos et al., 2018; Alcorn et al., 2019), and new benchmarks testing these systems has often found their performance to be much lower than both their ImageNet…”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “While the research community must create robust models that generalize to new scenarios, the robustness literature [6, 9] lacks consensus on evaluation benchmarks and contains many dissonant hypotheses.”
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Dodge & Karam (2017a) address underﬁtting via a mixture of corruption-speciﬁc experts assuming corruptions are known beforehand.”
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, background): “Furthermore, while ImageNet-trained CNNs generalise poorly towards a wide range of image distortions (e.g. Dodge & Karam, 2017; Geirhos et al., 2017; 2018), our ResNet-50 trained on Stylized-ImageNet often reaches or even surpasses human-level robustness (without ever being trained on the speciﬁc…”
Certified Adversarial Robustness via Randomized Smoothing (2019, methodology): “The robustness of neural networks to random noise has been analyzed both theoretically (Fawzi et al., 2016; Franceschi et al., 2018) and empirically (Dodge & Karam, 2017).”
Generalisation in humans and deep neural networks (2018, background): “We were able to not only close the human-DNN performance gap that was observed by [13] (who fine-tuned networks on distortions, reporting improved but not human-level DNN performance) but to surpass human performance in this respect.”
A Fourier Perspective on Model Robustness in Computer Vision (2019, background): “For example, it has been observed that commonly occuring image corruptions, such as random noise, contrast change, and blurring, can lead to signiﬁcant performance degradation (Dodge & Karam, 2017; Azulay However, data augmentation rarely improves robustness across all corruption types.”
Adversarial Examples Are a Natural Consequence of Test Error in Noise (2019, background): “For example, they are distinctly subhuman when classifying images distorted with additive Gaussian noise (Dodge & Karam, 2017), they lack robustness to different types of blur, pixelation, and changes in brightness (Hendrycks & Dietterich, 2018), lack robustness to random translations of the input…”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “For example, neural networks are sensitive to small translations and changes in scale [2], blurring and additive noise [10], small objects placed in images [41], and even different images from a similar distribution of the training set [40, 39].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Second, common perturbations such as blurriness or shifts in illumination, tend to reduce performance (Dodge and Karam, 2017; Temel et al., 2018; Hendrycks and Dietterich, 2019).”
Biologically inspired sleep algorithm for artificial neural networks (2019, background): “In contrast, humans robustly act in the presence of limited or altered (e.g., by noise) stimulus conditions [5, 6].”
Tradeoffs in Data Augmentation: An Empirical Study (2021, methodology): “This same metric has been used as a measure of a model’s robustness to image corruptions that do not change images’ semantic content (Azulay & Weiss, 2018; Dodge & Karam, 2017; Ford et al., 2019; Hendrycks & Dietterich, 2019; Rosenfeld et al., 2018; Yin et al., 2019).”
A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions (2020, background): “Two recent studies compare humans and DNNs on recognizing corrupted images, showing that DNN performance drops much faster than human performance for increased perturbation sizes [5], [10].”
Partial success in closing the gap between human and machine vision (2021, background): “For a long time, CNNs were unable to match human robustness in terms of generalisation beyond the training distribution—a large OOD robustness gap [14, 21, 39, 40].”
Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI (2018, background): “In recent years, not only have deep learning algorithms been able to surpass performance of other methods in artiﬁcial intelligence [1] but in some tasks, they have shown performance superior to humans [2, 3, 4].”
Increasing the robustness of DNNs against image corruptions by playing the Game of Noise (2020, methodology): “Two recent studies compare humans and DNNs on recognizing corrupted images, showing that DNN performance drops much faster than human performance for increased perturbation sizes [Dodge and Karam, 2017a, Geirhos et al., 2018].”
Comparing deep neural networks against humans: object recognition when the signal gets weaker (2017, result): “…time, ﬁne-tuning on speciﬁc conditions does not seem to generalize well to other conditions (e.g. ﬁne-tuning on uniform noise does not improve performance for salt-and-pepper noise), a ﬁnding consistent with results by Dodge and Karam (2017) who examined the impact of ﬁne-tuning on noise and blur.”
From ImageNet to Image Classification: Contextualizing Progress on Benchmarks (2020, methodology): “An MTurk study using a subset of the ImageNet classes is presented in Dodge & Karam (2017).”