Learning Continuous Attractors in Recurrent Networks

By Sebastian Seung. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Deep Generative Stochastic Networks Trainable by Backprop (2013, background): “…graph allows information to propagate both upwards (from input to higher levels) and downwards, giving rise to the computational graph structure illustrated in Figure 3, which is similar to that explored for deterministic recurrent auto-encoders (Seung, 1998; Behnke, 2001; Savard, 2011).”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “is also related to attractor network [61] with the Langevin dynamics serving as the attractor dynamics.”
A Theory of Generative ConvNet (2016, background): “Proposition 2 is interestingly related to the Hopfield network [11] and attractor network [30].”
Representation Learning: A Review and New Perspectives (2012, methodology): “An early approach (Jain and Seung, 2008) trained a standard but deep convolutional MLP on the task of denoising images, i.e. as a deep, convolutional, denoising auto-encoder.”
Recent advances in convolutional neural networks (2015, background): “oposed [85, 86, 94]. Conditional VAE [86, 94] generates samples from the conditional distribution with x~ p (x jy;z ). Denoising VAE [85] recovers the original input x from the corrupted input ^ x [90, 95]. Jensen-Shannon Divergence (JSD) is a symmetrical form of KLD. It measures the similarity bet”
Cooperative Training of Descriptor and Generator Networks (2016, background): “The Langevin dynamics in Step D1 can be considered an attractor dynamics toward the local modes for memory recall [48], while Step D2 shifts the local modes towards the observed examples.”
Denoising Criterion for Variational Auto-Encoding Framework (2015, methodology): “With the denoising autoencoder criterion (Seung 1998; Vincent et al. 2008), the input is corrupted according to some noise distribution, and the model needs to learn to reconstruct the original input (e.”
A working memory model based on fast Hebbian learning (2003, result): “This is similar to the models of eye position control and invariant object recognition described by Seung [44, 45] where line attractors are formed through learning.”
Natural Image Denoising with Convolutional Networks (2008, methodology): “It will be interesting to combine the two approaches to learn models that are " unnormalized priors " in the sense of energy-based image attractors ; regression can then be used as a tool for unsupervised learning by capturing dependencies between variables within the same distribution [22].”
Learning Iterative Image Reconstruction in the Neural Abstraction Pyramid (2001, background): “Continuous attractor networks have been proposed to complete images with occlusions [Seung, 1998].”
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion (2010, background): “The approach clearly has roots in their earlier work (Seung, 1998) and appears also inspired by more recent research on deep network pretraining, including our own group’s (Bengio et al., 2007).”
Training Products of Experts by Minimizing Contrastive Divergence (2002, background): “There have been previous attempts to learn representations by adjusting parameters to cancel out the effects of brief iteration in a recurrent network (Hinton & McClelland, 1988; O’Reilly, 1996; Seung, 1998 ), but these were 1794 Geoffrey E. Hinton”
Face localization and tracking in the neural abstraction pyramid (2005, unknown): “The network dynamics can be viewed as being settled into a continuous attractor [14], where input chang es along a manifold (in this case translations) make little dif ference to the energy of the system.”
Discriminative Recurrent Sparse Auto-Encoders (2013, background): “Finally, they are similar to the recurrent networks described by Seung (1998), but have recurrent connections amongst the hidden units, rather than between the hidden units and the input units, and introduce classification and sparsification losses.”
Scheduled denoising autoencoders (2014, background): “data set among permutation-invariant methods. 2 BACKGROUND The core idea of learning a representation by learning to reconstruct artiﬁcially corrupted training data dates back at least to the work of Seung (1998), who suggested using a recurrent neural network for this purpose. Using unsupervised la”
A Connection Between Score Matching and Denoising Autoencoders (2011, methodology): “Denoising autoencoders have also been used in different contexts in the earlier works of LeCun (1987); Gallinari, LeCun, Thiria, and Fogelman-Soulie (1987); and Seung (1998).”