Partial success in closing the gap between human and machine vision
By Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix Wichmann, Wieland Brendel. Neural Information Processing Systems, 2021
298 citations recorded by the source corpus.
A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines"in the wild"and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B). Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/
Subfields: Artificial Intelligence, Cognitive Neuroscience
View Paper: https://www.semanticscholar.org/paper/0e100c06d9fbdf32c434fd40469939a4aaab6c24
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Houlsby
- Quoc V. Le
- Yoshua Bengio
- Xiaohua Zhai
- Mostafa Dehghani
- Dacheng Tao
- Lucas Beyer
- Jianfeng Gao
In Cognitive Neuroscience
- Surya Ganguli
- Ari S. Morcos
- Shashank Shekhar
- Ben Sorscher
- Michael Tschannen
- Yi Tay
- Alexander Kolesnikov
- Shahaf E. Finder
Citations in the corpus
- Wavelet Convolutions for Large Receptive Fields (2024, methodology): “We use the modelvshuman benchmark [18] to quantify the improvement in shape bias ( i.e ., the fraction of predictions made based on shapes rather than on textures).”
- Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “[59, 60] generated different OOD datasets and tested with classic deep models and human observers to record the errors that they made on these datasets.”
- Beyond neural scaling laws: beating power law scaling via data pruning (2022, methodology): “To this end, we use the model-vshuman toolbox [42] based on data and analyses from [43, 44, 45, 46, 47].”
- Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “Using the model-vs-human toolbox (Geirhos et al., 2021), we evaluate three ViT-22B models fine-tuned on ImageNet with different resolutions (224, 384, 560).”