The Intrinsic Dimension of Images and Its Impact on Learning

By Phillip E. Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, Tom Goldstein. 2021

413 citations recorded by the source corpus.

It is widely believed that natural image data exhibits low-dimensional structure despite the high dimensionality of conventional pixel representations. This idea underlies a common intuition for the remarkable success of deep learning in computer vision. In this work, we apply dimension estimation tools to popular datasets and investigate the role of low-dimensional structure in deep learning. We find that common natural image datasets indeed have very low intrinsic dimension relative to the high number of pixels in the images. Additionally, we find that low dimensional datasets are easier for neural networks to learn, and models solving these tasks generalize better from training to test data. Along the way, we develop a technique for validating our dimension estimation tools on synthetic data generated by GANs allowing us to actively manipulate the intrinsic dimension by controlling the image generation process. Code for our experiments may be found here https://github.com/ppope/dimensions.

Subfields: Artificial Intelligence, Media Technology, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Axel Sauer
2. Katja Schwarz
3. Srdjan Ostojic
4. Mehrdad Jazayeri
5. Andreas Geiger
6. Simon Lacoste-Julien
7. Sebastian Nowozin
8. Jun-Yan Zhu

In Media Technology

1. Axel Sauer
2. Andreas Geiger

In Radiology, Nuclear Medicine and Imaging

1. Axel Sauer
2. Andreas Geiger

In Representation & Self-Supervised Learning

1. Katja Schwarz
2. Axel Sauer
3. Andreas Geiger

In Statistical Learning Theory

1. Srdjan Ostojic

Citations in the corpus

• StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “Recent findings indicate that the intrinsic dimension of natural image datasets is relatively low [Pope et al. 2021], ImageNet’s dimension estimate is around 40.”
• Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity (2021, background): “The intrinsic dimensionality for more natural stimuli has not yet been systematically quantified, but it is likely that it would increase with the complexity of spatial [28] and/or temporal [29,30] dependencies.”