Learning Complex, Extended Sequences Using the Principle of History Compression

By J. Urgen Schmidhuber. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Training Very Deep Networks (2015, methodology): “A related recent technique is based on using soft targets from a shallow teacher network to aid in training deeper student networks in multiple stages [25], similar to the neural history compressor for sequences, where a slowly ticking teacher recurrent net is “distilled” into a quickly ticking stud”
Wide Residual Networks (2016, background): “We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts.”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “M approach collapses C and M into a single network, and uses PowerPlay-like [82, 91] behavioural replay (where the behaviour of a teacher net is compressed into a student net [79]) to avoid forgetting old prediction and control skills when learning new ones.”
Recurrent Highway Networks (2016, background): “Let x [ t ] ∈ R m and y [ t ] ∈ R n represent the output of a standard RNN at time t , W ∈ R n × m and R ∈ R n × n the input and recurrent weight matrices and b ∈ R n a bias vector.”
How to Construct Deep Recurrent Neural Networks (2013, background): “An RNN may be extended deeper in yet another way by stacking multiple recurrent hidden layers on top of each other (Schmidhuber, 1992; El Hihi and Bengio, 1996; Jaeger, 2007; Graves, 2013).”
Gated Feedback Recurrent Neural Networks (2015, methodology): “…vector h t which is a function of all the previous symbols x 1 , · · · , x t − 1 and current symbol x t : This approach of using a neural network to model a probability distribution over sequences is widely used, for instance, in language modeling (see, e.g., Bengio et al., 2001; Mikolov, 2012).”
A Clockwork RNN (2014, methodology): “The Sequence Chunker , Neural History Compressor or Hierarchical Temporal Memory (Schmidhuber, 1991; 1992) consists of a hierarchy or stack of RNN that may run at different time scales, but, unlike the simpler CW-RNN, it requires unsupervised event predictors: a higher-level RNN receives an input on”
Model Fusion via Optimal Transport (2019, neutral): “Another line of work by Hinton et al. [8], Bucilu ˇ a et al. [9], Schmidhuber [10] proposes distillation techniques.”
Deep learning in neural networks: An overview (2014, methodology): “1991: UL-based history compression through a deep stack of RNNs Aworking Very Deep Learner (Section 3) of 1991 (Schmidhuber, 1992b, 2013a) could perform credit assignment across hundreds of nonlinear operators or neural layers, by using unsupervised pretraining for a hierarchy of RNNs.”
On the Binding Problem in Artificial Neural Networks (2020, background): “Alternatively, Bengio (2019) argues for the importance of a low-dimensional ‘conscious state’ (working memory) composed of largely independent units of abstraction that can be selected via attention (perhaps reminiscient of Schmidhuber, 1992b).”
World Models (2018, methodology): “2012) behavioural replay (where the behaviour of a teacher net is compressed into a student net (Schmidhuber, 1992)) to avoid forgetting old prediction and control skills when learning new ones.”
Lifelong learning of human actions with deep neural network self-organization (2017, background): “On the other hand, this type of processing 625 requires pre-established gate values for overcoming the vanishing gradient problem of recurrent networks that implement backpropagation through time (Schmidhuber, 1992).”
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model (2016, background): “The idea of having part of the model running at different clock rates is related to multi-scale RNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutnik et al., 2014; Sordoni et al., 2015; Serban et al., 2016).”
Tagger: Deep Unsupervised Perceptual Grouping (2016, background): “Since recurrent neural networks (RNNs) are general purpose computers, they can in principle implement arbitrary computable types of temporary variable binding [25, 26], unsupervised segmentation [24], and internal [26] and external attention [28].”
Neural Expectation Maximization (2017, background): “Unsupervised Segmentation has been studied in several different contexts [30], from random vectors [14] over texture segmentation [10] to images [18, 16].”
Nematus: a Toolkit for Neural Machine Translation (2017, background): “This way of combining RNN blocks is similar to what is referred in the literature as deep transition RNNs (Pascanu et al., 2014; Zilly et al., 2016) as opposed to the more common stacked RNNs (Schmidhuber, 1992; El Hihi and Bengio, 1995; Graves, 2013). deep output Given s j , y j − 1 , and c j , the”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “, through predictive coding [228, 248]) videos of people placing toys in boxes, or to summarize such videos through textual outputs.”