Controlling Recurrent Neural Networks by Conceptors

By Herbert Jaeger. 2014

130 citations recorded by the source corpus.

The human brain is a dynamical system whose extremely complex sensor-driven neural processes give rise to conceptual, logical cognition. Understanding the interplay between nonlinear neural dynamics and concept-level cognition remains a major scientific challenge. Here I propose a mechanism of neurodynamical organization, called conceptors, which unites nonlinear dynamics with basic principles of conceptual abstraction and logic. It becomes possible to learn, store, abstract, focus, morph, generalize, de-noise and recognize a large number of dynamical patterns within a single neural system; novel patterns can be added without interfering with previously acquired ones; neural noise is automatically filtered. Conceptors help explaining how conceptual-level information processing emerges naturally and robustly in neural systems, and remove a number of roadblocks in the theory and applications of recurrent neural networks.

Subfields: Artificial Intelligence, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

Citations in the corpus

Continual learning with hypernetworks (2019, neutral): “This interpretation is consistent with a growing body of work which suggests the involvement of modulatory inputs in implementing context- or task-dependent network mode-switching [29, 30, 31].”
Continual Learning for Sentence Representations Using Conceptors (2019, background): “As a future direction, we expect to lift this assumption, for example, by updating the common direction statistics at a sentence level us-ing Autoconceptors (Jaeger, 2014, section 3.14).”
Continual learning of context-dependent processing in neural networks (2018, methodology): “This iterative method is related to the Recursive Least Square (RLS) algorithm [16, 17] (see Supplementary Information for the discussion), which can be used to train feedforward and recurrent neural networks to achieve fast convergence [18, 19], tame chaotic activities [20] and avoid interference b”
Overcoming Catastrophic Interference by Conceptors (2017, methodology): “A comprehensive treatment can be found in [Jaeger, 2014].”
Using Conceptors to Manage Neural Long-Term Memories for Temporal Patterns (2017, methodology): “To keep the length of this article within reasonable bounds, I have omitted a sizable portion of LTM-relevant material from Jaeger (2014), namely the topic of Boolean operations on conceptors and the options that such Boolean operations grant for incremental and differential memory extensibility…”
Caching in the Sky: Proactive Deployment of Cache-Enabled Unmanned Aerial Vehicles for Optimized Quality-of-Experience (2016, methodology): “The accuracy of ESN prediction is measured by normalized root mean square error [31].”
Single-label and multi-label conceptor classifiers in pre-trained neural networks (2018, methodology): “Conceptors were introduced in [17] in a context of recurrent neural networks as filters for recurrent states.”
Conceptor Debiasing of Word Representations Evaluated on WEAT (2019, methodology): “, debiasing) using the debiasing conceptor, a clean mathematical representation of subspaces that can be operated on and composed by logic-based manipulations (Jaeger, 2014).”
Unsupervised Post-processing of Word Vectors via Conceptor Negation (2018, methodology): “These operations all have interpretation on the data level, i.e., on the distribution of the random variable x (details in (Jaeger 2014, Section 3.9)).”
Machine Learning for Wireless Networks with Artificial Intelligence: A Tutorial on Neural Networks (2017, background): “Conceptors defined in [172] enable an ESN to perform a large number of mobility and content request patterns predictions.”
The Role of Protected Class Word Lists in Bias Identification of Contextualized Word Representations (2019, methodology): “As described in the orignal work on matrix conceptors (Jaeger, 2014; He and Jaeger, 2018; Liu et al., 2019b,a) C has a closed form solution: Intuitively, C is a soft projection matrix on the linear subspace that gives the largest shrinkage where the word embeddings Z have the highest variance.”
Neural-Symbolic Learning and Reasoning: A Survey and Interpretation (2017, background): “We first want to direct the reader’s attention to a recently proposed neuro-computational mechanism called “conceptors” (Jaeger, 2014).”
Meta Continual Learning (2018, methodology): “Using conceptors originally proposed in [9], He and Jaeger [8] adjusted the gradients given by backpropagation so that learning a new task would minimally interfere with previously learned tasks.”