Recurrence is required to capture the representational dynamics of the human visual system

By Tim C Kietzmann, Courtney J. Spoerer, Lynn K. A. Sörensen, Radoslaw Martin Cichy, Olaf Hauk, Nikolaus Kriegeskorte. 2019

405 citations recorded by the source corpus.

Significance Understanding the computational principles that underlie human vision is a key challenge for neuroscience and could help improve machine vision. Feedforward neural network models process their input through a deep cascade of computations. These models can recognize objects in images and explain aspects of human rapid recognition. However, the human brain contains recurrent connections within and between stages of the cascade, which are missing from the models that dominate both engineering and neuroscience. Here, we measure and model the dynamics of human brain activity during visual perception. We compare feedforward and recurrent neural network models and find that only recurrent models can account for the dynamic transformations of representations among multiple regions of visual cortex. The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time-resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral-stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform parameter-matched feedforward models in terms of their ability to capture the multiregion cortical dynamics. Targeted virtual cooling experiments on the recurrent deep network models further substantiate the importance of their lateral and top-down connections. These results establish that recurrent models are required to understand information processing in the human ventral stream.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Model Zoo: A Growing Brain That Learns Continually (2021, background): “…datum only once is well-taken, one single epoch is quite insuﬃcient to train modern deep networks; if one thinks of biological considerations, local-descent algorithms like stochastic gradient descent (SGD) are quite diﬀerent from recurrent circuits in the biological brain (Kietzmann et al., 2019).”
Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. (2020, methodology): “already being used to focus experimental resources on the most interesting aspects of brain function that are not yet accurately described (Tang et al., 2018; Kar et al., 2019; Hénaff et al., 2019; Kietzmann et al., 2019; Golan et al., 2019).”
GLiT: Neural Architecture Search for Global and Local Image Transformer (2021, background): “Second, local recurrence in human visual system [12, 13] is not realized in the transformers like ViT and DeiT.”
High-resolution image reconstruction with latent diffusion models from human brain activity (2023, methodology): “…and design of deep neural network models, have allowed direct comparisons be-tween latent representations in biological brains and architectural characteristics of artiﬁcial networks, providing important insights into how these systems operate [3, 8– 10, 13, 18, 19, 21, 42, 43, 54, 55].”
Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency (2020, background): “[54] T.”
If deep learning is the answer, what is the question? (2020, background): “For example, adding recurrent connections improves the fit to neural data 18 , especially for those natural images that are harder to classify and at later poststimulus time points 17 , whereas including a biologically plausible front end (a ‘retina net’) encourages the formation of realistic…”
Individual differences among deep neural network models (2020, background): “Although further experiments are required, we expect our results to generalize to representations learned by (unrolled) recurrent neural network architectures 25,26 , if not explicitly constrained 27 .”
Artificial Neural Networks for Neuroscientists: A Primer (2020, background): “In convolutional recurrent networks, neurons in each layer further receive feedback inputs from later layers and local recurrent connections (Nayebi et al., 2018; Kietzmann et al., 2019).”