A parametric texture model based on deep convolutional features closely matches texture appearance for humans

By Thomas S. A. Wallis, Christina M. Funke, Alexander S. Ecker, Leon A. Gatys, Felix Wichmann, Matthias Bethge. 2017

63 citations recorded by the source corpus.

Our visual environment is full of texture—“stuff” like cloth, bark or gravel as distinct from “things” like dresses, trees or paths—and humans are adept at perceiving subtle variations in material properties. To investigate image features important for texture perception, we psychophysically compare a recent parameteric model of texture appearance (CNN model) that uses the features encoded by a deep convolutional neural network (VGG-19) with two other models: the venerable Portilla and Simoncelli model (PS) and an extension of the CNN model in which the power spectrum is additionally matched. Observers discriminated model-generated textures from original natural textures in a spatial three-alternative oddity paradigm under two viewing conditions: when test patches were briefly presented to the near-periphery (“parafoveal”) and when observers were able to make eye movements to all three patches (“inspection”). Under parafoveal viewing, observers were unable to discriminate 10 of 12 original images from CNN model images, and remarkably, the simpler PS model performed slightly better than the CNN model (11 textures). Under foveal inspection, matching CNN features captured appearance substantially better than the PS model (9 compared to 4 textures), and including the power spectrum improved appearance matching for two of the three remaining textures. None of the models we test here could produce indiscriminable images for one of the 12 textures under the inspection condition. While deep CNN (VGG-19) features can often be used to synthesise textures that humans cannot discriminate from natural textures, there is currently no uniformly best model for all textures and viewing conditions.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, background): “Intriguingly, this offers an explanation for a number of rather disconnected findings: CNNs match texture appearance for humans (Wallis et al., 2017), and their predictive power for neural responses along the human ventral stream appears to be largely due to human-like texture representations, but n”
Generalisation in humans and deep neural networks (2018, methodology): “Additionally, for experiments with training on distortions, we also evaluated performance on stimuli with salt-and-pepper noise (controlled by parameter p indicating probability of setting a pixel to either black or white; p ∈ [0 , 10 , 20 , 35 , 50 , 65 , 80 , 95]% ).”
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks (2018, background): “…recognition behaviors (Rajalingham et al., 2015; Jozwik et al., 2016; Kheradpisheh et al., 2016; Kubilius et al., 2016; Peterson et al., 2016; Wallis et al., 2017), and the internal hidden neurons of those same models also predict a large fraction of the image-driven response variance of…”
One-shot Texture Segmentation (2018, background): “There is evidence that outside of the foveal region the visual system represents image patches as textures [44, 10] which are robust to local shifts and deformations.”
Correspondence of Deep Neural Networks and the Brain for Visual Textures (2018, background): “Apart from their representation in V2, there is a rich history of studying texture representation in higher visual areas such as V4 [21], [22], [23], [24], [25], [26], and texture perception in psychophysics [27], [28], [29], [30], [31], [32].”