Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields

By Karol Gregor, Yann LeCun. 2010

73 citations recorded by the source corpus.

We introduce a new neural architecture and an unsupervised algorithm for learning invariant representations from temporal sequence of images. The system uses two groups of complex cells whose outputs are combined multiplicatively: one that represents the content of the image, constrained to be constant over several consecutive frames, and one that represents the precise location of features, which is allowed to vary over time but constrained to be sparse. The architecture uses an encoder to extract features, and a decoder to reconstruct the input from the features. The method was applied to patches extracted from consecutive movie frames and produces orientation and frequency selective units analogous to the complex cells in V1. An extension of the method is proposed to train a network composed of units with local receptive field spread over a large image of arbitrary size. A layer of complex cells, subject to sparsity constraints, pool feature units over overlapping local neighborhoods, which causes the feature units to organize themselves into pinwheel patterns of orientation-selective receptive fields, similar to those observed in the mammalian visual cortex. A feed-forward encoder efficiently computes the feature representation of full images.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Deep AutoRegressive Networks (2013, methodology): “In this paper we use the periodic local connectivity of Gregor & LeCun (2010) which uses less weight sharing than full convolutional networks.”
Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (2016, background): “The works of [11, 7] introduced the concept of constructing a local receptive field to reduce the number of learned parameters.”
Spectral Networks and Locally Connected Networks on Graphs (2013, methodology): “Using arbitrary ﬁlters instead of generic fully connected layers reduces the complexity to O ( n ) parameters per feature map, as does using the metric structure by building a “locally connected” net [8, 17].”
Deep Convolutional Networks on Graph-Structured Data (2015, background): “There have been several works which have explored architectures using the so-called local receptive ﬁelds [6, 4, 14], mostly with applications to image recognition.”
Predicting Parameters in Deep Learning (2013, background): “This has parallels to other work on inducing topology in representations [10] as well as work on learning pooling structures in deep networks [4].”
Building high-level features using large scale unsupervised learning (2011, methodology): “Next, to achieve invariance to local deformations, we employ local L2 pooling (Hyvärinen et al., 2009; Gregor & LeCun, 2010; Le et al., 2010) and local contrast normalization (Jarrett et al., 2009).”
Video (language) modeling: a baseline for generative models of natural videos (2014, methodology): “Wiskott & Sejnowski (2002) have proposed instead a method based on slowness of features through time, an idea later extended in a bilinear model by Gregor & LeCun (2010).”
Structured sparse coding via lateral inhibition (2011, background): “We also bind connections weights for units that are far enough from each other to reduce the number of parameters without affecting the local structure (Gregor and LeCun, 2010).”
Texture Modeling with Convolutional Spike-and-Slab RBMs and Deep Extensions (2012, methodology): “Following the recent exploration of Boltzmann machines for textures (Kivinen and Williams, 2012), we have trained ssRBMs with tiled-convolution weight sharing Gregor and LeCun (2010); Le et al. (2010) on the Brodatz-texture images1.”
Dynamic Region-Aware Convolution (2020, methodology): “utilize the semantic information in image data, local convolution [9] applies individual unshared filters to each pixel, which has great potential in tasks that don’t require translation-invariance.”
DeepFace: Closing the Gap to Human-Level Performance in Face Verification (2014, background): “The subsequent layers (L4, L5 and L6) are instead locally connected [13, 16], like a convolutional layer they apply a filter bank, but every location in the feature map learns a different set of filters.”
Tiled convolutional neural networks (2010, unknown): “[7] K.”
Selecting Receptive Fields in Deep Networks (2011, methodology): “These methods have many advantages but require us to specify a topography first, then solve a large-scale optimization problem in order to organize our features according to the given topographic layout.”
Generating more realistic images using gated MRF's (2010, methodology): “Following Gregor and LeCun [18] and also Tang and Eliasmith [19], we keep the number of parameters under control by using local potentials, but unlike Roth and Black [6] we only share weights between potentials that do not overlap.”
SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-Grained Recognition (2016, background): “Note that our pfc layer is different from other works [17, 20, 35], where the local convolutional filters are utilized to specified local regions.”