Memory traces in dynamical systems

By Surya Ganguli, Dongsung Huh, Haim Sompolinsky. Proceedings of the National Academy of Sciences of the United States of America, 2008

327 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/80c77d57a4e1c365d1feb7c0a1ba165b10f7de7c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

Citations in the corpus

Capacity and Trainability in Recurrent Neural Networks (2016, background): “Previous work on short term memory traces explores the tradeoffs between memory fidelity and duration, for the case that a new input is presented to the RNN at every time step (Jaeger & Haas, 2004; Maass et al., 2002; White et al., 2004; Ganguli et al., 2008; Charles et al., 2014).”
Physical reservoir computing—an introductory perspective (2020, background): “Focusing on ESN, the behaviors of memory capacity and their related measures are studied in detail with a linear activation function [46–51] and, recently, with a nonlinear activation function [52–54].”
A theory of multineuronal dimensionality, dynamics and measurement (2017, methodology): “…has dimensionality D much less that because the connectivity is designed to amplify activity in a small number of dimensions and attenuate activity in all others, similar to the way non-normal networks have been proposed to play a functional role in sequence memory [Ganguli et al., 2008b].”
Memory without Feedback in a Neural Network (2009, background): “…tasks ranging from speech recognition to EEG analysis (Principe et al., 1993; Tank and Hopﬁeld, 1987), and a recent information-theoretic study (Ganguli et al., 2008b) shows that the information remembered by a network about a previous input signal is maximal for a network with functionally…”
On simplicity and complexity in the brave new world of large-scale neuroscience. (2015, background): “In the realm of short-term sequence memory, the idea of liquid state machines [80, 81] posited that generic neural circuits could convert temporal information into instantaneous spatial patterns of activity, but theoretical work [82, 83, 84] revealed general principles relating circuit connectivity ”
Generating Coherent Patterns of Activity from Chaotic Neural Networks (2009, unknown): “Ganguli et al. (2008) have discussed the advantages of using an effective delay-line architecture in applications of networks to memory retention.”
State-dependent computations: spatiotemporal processing in cortical networks (2009, background): “Additional theoretical work shows that under certain conditions randomly connected networks can encode past information in the ongoing dynamics of the active states, and the duration of this fading memory increases with network siz”
Recurrent Orthogonal Networks and Long-Memory Tasks (2016, background): “The authors of (White et al., 2004) and (Ganguli et al., 2008) study the ability of linear RNNs (with orthogonal and generic transition matrices, respectively) to store scalar sequences in their hidden state, and show that the memory capacity scales with the number of hidden units.”
Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. (2012, background): “Hidden low-dimensional structure in neural activity patterns has been found in several systems (Ganguli et al. 2008a, Yu et al. 2009, Machens et al. 2010), and moreover, intrinsic spatiotemporal fluctuations exhibited in many models of recurrent neuronal circuits, including chaotic networks, are…”
Initialization matters: Orthogonal Predictive State Recurrent Neural Networks (2018, background): “Related results are presented in Henaff et al. (2016), Saxe et al. (2013) (orthogonal initialization for training acceleration), Ganguli et al. (2008) and White et al. (2004).”
Spiking activity propagation in neuronal networks: reconciling different perspectives on neural coding (2010, background): “Thus, even in a recurrent network, the flow of spiking activity on a local scale ('locality' being defined by connection delays, firing rates and refractoriness), can be treated as an FF”
Learning Attractor Dynamics for Generative Memory (2018, background): “In this paper, we focus on point attractors, although other types of attractor may also support memory systems [12].”
Statistical mechanics of complex neural systems and high dimensional data (2013, background): “Also diverse applications of these same techniques to neuronal information processing are discussed in [30], including semantic information processing [42, 43, 44], short-term memory [45, 46], neural circuits for L1 minimization [47], learning sparse representations [48, 49], regularized learning of”
Minimum Complexity Echo State Network (2011, background): “Of course, the optimal input weight considerations of [30] hold for linear reservoir models only.”
ROBUST TIMING AND MOTOR PATTERNS BY TAMING CHAOS IN RECURRENT NEURAL NETWORKS (2012, background): “4A shows that a network of size N =800 ( g =1.5) reliably learned a 5000 ms delay, but not a 6000 ms delay, reflecting the finite “memory” of such networks (Ganguli et al., 2008; Jaeger, 2001).”