Learning hierarchical category structure in deep neural networks

By Andrew M. Saxe, James L. McClelland, Surya Ganguli. 2013

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, methodology): “The training set consists of 32 orthogonal input patterns, each associated with a 1000-dimensional feature vector generated by a hierarchical diffusion process described in [16] with a five level binary tree and flip probability of 0.”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, background): “They qualitatively recapitulate aspects of the hierarchical development of semantic concepts in infants (Saxe et al. , 2013).”
An analytic theory of generalization dynamics and transfer learning in deep linear networks (2018, methodology): “The results in this section assume a single hidden layer, but Saxe et al. (2013a) derived t ( s, ˆ s ) for networks of arbitrary depth and we apply our theory to some deeper networks.”
On simplicity and complexity in the brave new world of large-scale neuroscience. (2015, background): “The theory of deep learning is still in its infancy, but some examples of general statements one can make about deep circuits without actually simulating them include how their functional complexity scales with depth [70, 71], how their synaptic weights, over time, acquire statistical structures in ”
Unsupervised Learning of Visual Structure using Predictive Generative Networks (2015, background): “The sequence in which the model learns is reminiscent of theoretical work supporting the notion that modes in the dataset are learned in a coarse-to-ﬁne fashion (Saxe et al., 2013).”
Efficiency of learning vs. processing: Towards a normative theory of multitasking (2020, background): “Moreover, learning dynamics in neural networks themselves promote the learning of shared representation based on shared structure in the task environment (Hinton, 1986; Saxe, McClelland, & Ganguli, 2013; Mus-slick et al., 2017).”
On the saddle point problem for non-convex optimization (2014, background): “These learning dynamics exhibit plateaus of high error followed by abrupt transitions to better performance, and they qualitatively recapitulate aspects of the hierarchical development of semantic concepts in infants (Saxe et al. , 2013).”
Multitasking Capability Versus Learning Efficiency in Neural Network Architectures (2017, background): “The simulation results conﬁrm well-known explorations in neural networks (Hinton, 1986; McClelland & Rogers, 2003; Saxe et al., 2013) that task similarities in the environment can translate into similarities between learned task representations.”
Local minima in training of deep networks (2017, unknown): “This intuition is carried further by Saxe et al. (2014; 2013) where deep linear models are studied.”
Local minima in training of neural networks (2016, result): “Extensions of these early results can be found in Saxe et al. (2014; 2013).”