Slowness and Sparseness Have Diverging Effects on Complex Cell Learning

By Jörn-Philipp Lies, Ralf M. Häfner, Matthias Bethge. 2014

26 citations recorded by the source corpus.

Following earlier studies which showed that a sparse coding principle may explain the receptive field properties of complex cells in primary visual cortex, it has been concluded that the same properties may be equally derived from a slowness principle. In contrast to this claim, we here show that slowness and sparsity drive the representations towards substantially different receptive field properties. To do so, we present complete sets of basis functions learned with slow subspace analysis (SSA) in case of natural movies as well as translations, rotations, and scalings of natural images. SSA directly parallels independent subspace analysis (ISA) with the only difference that SSA maximizes slowness instead of sparsity. We find a large discrepancy between the filter shapes learned with SSA and ISA. We argue that SSA can be understood as a generalization of the Fourier transform where the power spectrum corresponds to the maximally slow subspace energies in SSA. Finally, we investigate the trade-off between slowness and sparseness when combined in one objective function.

Subfields: Biophysics, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

1. Kristen Grauman
2. Dinesh Jayaraman
3. Yann LeCun
4. David Eigen
5. Limin Wang
6. Tianhao Li

In Cognitive Neuroscience

1. Joan Bruna
2. Dinesh Jayaraman
3. Yann LeCun
4. Kristen Grauman
5. David Eigen
6. Ross Goroshin
7. Jonathan Tompson

Citations in the corpus

• Unsupervised Learning of Spatiotemporally Coherent Metrics (2014, methodology): “In the work by [17], the linear part of the transformation into feature space is constrained to be unitary.”
• Learning Image Representations Tied to Egomotion from Unlabeled Video (2017, methodology): “Typically, one seeks invariance to small transformations, e.g., the orientation binning and pooling operations in SIFT/HOG (Lowe 1999; Dalal and Triggs 2005; Tuytelaars andMikolajczyk 2008) andmodern CNNs both target invariance to local translations and rotations.”
• Learning Image Representations Tied to Ego-Motion (2015, background): “Temporal coherence has been explored for unsupervised feature learning with CNNs [22, 37, 9, 3, 19], with applications to dimensionality reduction [10], object recognition [22, 37], and metric learning [9].”
• Diverse feature visualizations reveal invariances in early layers of deep neural networks (2018, unknown): “This energy model has also been used to study rotation, scaling and more general invariances in the context of unsupervised representation learning [2, 3, 15] An alternative formulation was originally proposed by Hubel & Wiesel, who discovered complex cells in the 1960ies in the primary visual corte”
• Learning image representations equivariant to ego-motion (2015, background): “Temporal coherence has been explored for unsupervised feature learning with CNNs [20] [35] [7] [3] [17], with applications to dimensionality reduction [8], object recognition [20] [35], and metric learning [7].”