Neuronal Synchrony in Complex-Valued Deep Networks

By David P. Reichert, Thomas Serre. 2013

121 citations recorded by the source corpus.

Deep learning has recently led to great successes in tasks such as image recognition (e.g Krizhevsky et al., 2012). However, deep networks are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to cortical circuits. The challenge is to identify which neuronal mechanisms are relevant, and to find suitable abstractions to model them. Here, we show how aspects of spike timing, long hypothesized to play a crucial role in cortical information processing, could be incorporated into deep networks to build richer, versatile representations. We introduce a neural network formulation based on complex-valued neuronal units that is not only biologically meaningful but also amenable to a variety of deep learning frameworks. Here, units are attributed both a firing rate and a phase, the latter indicating properties of spike timing. We show how this formulation qualitatively captures several aspects thought to be related to neuronal synchrony, including gating of information processing and dynamic binding of distributed object representations. Focusing on the latter, we demonstrate the potential of the approach in several simple experiments. Thus, neuronal synchrony could be a flexible mechanism that fulfills multiple functional roles in deep networks.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Successful integrations include complex-valued Boltzmann Machines (Reichert and Serre, 2014; Zemel et al., 1995) and complex-valued RNNs that could be trained either with backpropagation (Mozer et al.”
Tagger: Deep Unsupervised Perceptual Grouping (2016, methodology): “Rather, complex values have been used to model oscillating activations using the phase as soft tags for synchronization [18, 20].”
Neural Expectation Maximization (2017, methodology): “Later, complex valued activations have been used to replace the explicit simulation of oscillation [25, 26].”
Binding via Reconstruction Clustering (2015, background): “Recently, these ideas were implemented using complex valued activations in neural networks to jointly encode ﬁring rate and phase (Rao et al., 2008; Reichert & Serre, 2013).”
Encoding word order in complex embeddings (2019, methodology): “Complex-valued weights have been used in NNs, motivated by biology (Reichert & Serre, 2013), and also as signal processing in speech recognition (Shi et al., 2006).”
Generative Modeling of Infinite Occluded Objects for Compositional Scene Representation (2019, background): “(Wang & Terman, 1995; Rao et al., 2008; Reichert & Serre, 2014) use neuronal synchrony as the mechanism and segregate objects based on simulated phases of neurons.”
Deep Complex Networks (2017, background): “Reichert and Serre [2013] have proposed a biologically plausible deep network that allows one to construct richer and more versatile representations using complex-valued neuronal units.”