Self-Delimiting Neural Networks

By J. Urgen Schmidhuber. 2012

39 citations recorded by the source corpus.

Self-delimiting (SLIM) programs are a central concept of theoretical computer science, particularly algorithmic information & probability theory, and asymptotically optimal program search (AOPS). To apply AOPS to (possibly recurrent) neural networks (NNs), I introduce SLIM NNs. Neurons of a typical SLIM NN have threshold activation functions. During a computational episode, activations are spreading from input neurons through the SLIM NN until the computation activates a special halt neuron. Weights of the NN's used connections define its program. Halting programs form a prefix code. The reset of the initial NN state does not cost more than the latest program execution. Since prefixes of SLIM programs influence their suffixes (weight changes occurring early in an episode influence which weights are considered later), SLIM NN learning algorithms (LAs) should execute weight changes online during activation spreading. This can be achieved by applying AOPS to growing SLIM NNs. To efficiently teach a SLIM NN to solve many tasks, such as correctly classifying many different patterns, or solving many different robot control tasks, each connection keeps a list of tasks it is used for. The lists may be efficiently updated during training. To evaluate the overall effect of currently tested weight changes, a SLIM NN LA needs to re-test performance only on the efficiently computable union of tasks potentially affected by the current weight changes. Future SLIM NNs will be implemented on 3-dimensional brain-like multi-processor hardware. Their LAs will minimize task-specific total wire length of used connections, to encourage efficient solutions of subtasks by subsets of neurons that are physically close. The novel class of SLIM NN LAs is currently being probed in ongoing experiments to be reported in separate papers.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Adaptive Computation Time for Recurrent Neural Networks (2016, background): “A related architecture known as Self-Delimiting Neural Networks [26, 30] employs a halting neuron to end a particular update within a large, partially activated network; in this case however a simple activation threshold is used to make the decision, and no gradient with respect to halting time is p”
Recurrent Highway Networks (2016, background): “A known way of creating large step-by-step depth is to let an RNN tick for several "micro time steps” per step of the sequence [19–21], where the RNN could be an LSTM with forget gates.”
Deep learning in neural networks: An overview (2014, methodology): “…GPU-MPCNNs was the first system to achieve superhuman visual pattern recognition (Ciresan, Meier, Masci, Schmidhuber, 2011; Ciresan, Meier, Masci, & Schmidhuber, 2012) in a controlled competition, namely, the IJCNN 2011 traffic sign recognition contest in San Jose (CA) (Stallkamp, Schlipsing,…”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “5As an aside, the special checkout item connects to self-delimiting neural networks (Schmidhuber, 2012), where a “halt neuron” is used to signal the end of a sequence.”
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem (2011, neutral): “, input neurons of an NN [47]), and a unique description of a deterministic procedure for determining whether the problem has been solved.”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Self-organizing learners, disentanglement, hierarchical learning: Research on self-organizing learners [20, 70, 72, 73] has so far been relatively theoeretical; CRL is a synthesis of recent work in disentanglement and hierarchical learning that attempts to bring this old idea closer to reality.”
First Experiments with PowerPlay (2012, methodology): “As a concrete implementation of the solver, we use a special neural network (NN) [2] architecture called the Self-Delimiting NN or SLIM NN [25].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “But we can implement M as a self-modularizing, computation cost-minimizing, winner-take-all RNN [221, 242, 267].”
Modular Networks: Learning to Decompose Neural Computation (2018, background): “Conditional computation can also be achieved through activation sparsity or winner-take-all mechanisms [27, 23, 24] but is hard to parallelize on modern accelerators such as GPUs.”
One Big Net For Everything (2018, background): “vious traces of still relevant trials (the simplest option to achieve the POWERPLAY criterion [75]), one can also implement ONE as a selfmodularizing, computation cost-minimizing, winner-take-all RNN [53, 74, 86]. Then we can keep track of which weights of ONE are used for which tasks. That is, to t”
Understanding Locally Competitive Networks (2014, background): “Selfdelimiting recurrent networks with competitive units [3, 9] can in principle learn to decide their own run time and effective number of parameters, thus learning their own computable regularizers.”