Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior

By Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, James J. DiCarlo. Nature Neuroscience, 2018

488 citations recorded by the source corpus.

Non-recurrent deep convolutional neural networks (CNNs) are currently the best at modeling core object recognition, a behavior that is supported by the densely recurrent primate ventral stream, culminating in the inferior temporal (IT) cortex. If recurrence is critical to this behavior, then primates should outperform feedforward-only deep CNNs for images that require additional recurrent processing beyond the feedforward IT response. Here we first used behavioral methods to discover hundreds of these ‘challenge’ images. Second, using large-scale electrophysiology, we observed that behaviorally sufficient object identity solutions emerged ~30 ms later in the IT cortex for challenge images compared with primate performance-matched ‘control’ images. Third, these behaviorally critical late-phase IT response patterns were poorly predicted by feedforward deep CNN activations. Notably, very-deep CNNs and shallower recurrent CNNs better predicted these late IT responses, suggesting that there is a functional equivalence between additional nonlinear transformations and recurrence. Beyond arguing that recurrent circuits are critical for rapid object identification, our results provide strong constraints for future recurrent model development. Using model- and primate behavior-driven image selection with large-scale electrophysiology in monkeys performing core recognition tasks, Kar et al. provide evidence that automatically engaged recurrent circuits are critical for rapid object identification.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/749118e990a532dddf9f573676e9d178f8442e2d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

The Notorious Difficulty of Comparing Human and Machine Perception (2020, unknown): “…the enticing possibility that we may learn from one system by studying the other (Hassabis, Kumaran, Summerﬁeld, & Botvinick, 2017; Jozwik, Kriegeskorte, Cichy, & Mur, 2018; Kar, Kubilius, Schmidt, Issa, & DiCarlo, 2019; Kriegeskorte & Douglas, 2018; Lake, Ullman, Tenenbaum, & Gershman, 2017).”
If deep learning is the answer, what is the question? (2020, background): “…improves the fit to neural data 18 , especially for those natural images that are harder to classify and at later poststimulus time points 17 , whereas including a biologically plausible front end (a ‘retina net’) encourages the formation of realistic coding properties, including cell…”
Convolutional Bipartite Attractor Networks (2019, background): “Incorporating recurrent circuits into deep object recognition models not only improves their performance, but does so speciﬁcally for images that humans and animals take the longest to process [15].”
Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks (2021, background): “Iterative processes underlie the human ability to solve sequential reasoning problems, such as complex question answering, proof writing, and even object classiﬁcation [Liao and Poggio, 2016, Kar et al., 2019].”
Unsupervised neural network models of the ventral visual stream (2020, background): “However, while feedforward networks might be sufficient to predict temporal averages during the first volley of stimulus-evoked neural responses, they are insufficient to describe the response dynamics of real neurons (73).”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “These connections can also help make the CNN representations a better match to neural data, particularly for challenging images and at later time points in the response (Kar et al., 2019; Kubilius et al., 2019; Rajaei, Mohsenzadeh, Ebrahimpour, & KhalighRazavi, 2019; Shi, Wen, Zhang, Han, & Liu, 201”
Learning Physical Graph Representations from Visual Scenes (2020, background): “Such local recurrent and long-range feedback connections are ubiquitous in the primate visual system, where it is hypothesized that they play a role in object recognition and scene understanding [22, 46, 49].”
Artificial Neural Networks for Neuroscientists: A Primer (2020, background): “For example, compared to purely feedforward networks, recurrently-connected deep networks are better at predicting responses of higher visual area neurons to behaviorally challenging images [Kar et al., 2019].”
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019, background): “We identify that these results are primarily driven by recurrent connections, in line with our understanding of how the primate visual system processes visual information [42, 14].”
Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency (2020, background): “Recent evidence seems to indicate that recurrence may be especially useful for difficult images [57–59].”
Neural population control via deep image synthesis (2018, unknown): “With continuing improvement of the fidelity of ANN models of the ventral stream (1, 26, 27), the results presented here have likely only scratched the surface of what is possible with such implemented characterizations of the brain’s neural networks.”
Deep neuroethology of a virtual rodent (2019, background): “…functional transformations implemented by neurobiological circuits, prominently in vision (Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014; Kar et al., 2019), but also increasingly in other areas, including audition (Kell et al., 2018) and navigation (Banino et al., 2018; Cueva & Wei,…”
Recurrence is required to capture the representational dynamics of the human visual system (2019, background): “which, among other computational functions, may facilitate recognition under challenging conditions (6, 32, 33, 35).”