Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability

By J. Urgen Schmidhuber. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Weight Agnostic Neural Networks (2019, background): “Ideas related to MDL for making neural networks “simple” was proposed in the 1990s, such as simplifying networks by soft-weight sharing [80], reducing the amount of information in weights by making them noisy [43], and simplifying the search space of its weights [95].”
Deep learning in neural networks: An overview (2014, methodology): “A very general but not always practical approach for discovering low-complexity SL NNs or RL NNs searches among weight matrix-computing programs written in a universal programming language, with a bias towards fast and short programs (Schmidhuber, 1997) (Section 6.7).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “In very general early work, we used universal assembler-like languages to encode NNs [235], later coefficients of a Discrete Cosine Transform (DCT) [132].”
Optimal Ordered Problem Solver (2002, neutral): “Problem sequences that humans consider to be interesting are atypical when compared to arbitrary sequences of well-deﬁned problems (Schmidhuber, 1997).”
The Description Length of Deep Learning models (2018, methodology): “The compression viewpoint originally motivated the use of variational methods in neural networks (Hinton and Van Camp, 1993; Schmidhuber, 1997).”
One Big Net For Everything (2018, methodology): “The search space can often be reduced dramatically by evolving compact encodings of RNNs, e.g., [67, 87, 32, 94][76, Sec.”
Self-Delimiting Neural Networks (2012, methodology): “…more principled SLIM NNs can learn to actively select in task-speciﬁc ways their own size, that is, their effective number of weights ( = modiﬁable free parameters), in line with the theory of algorithmic probability and optimal universal inductive inference [71, 34, 54, 43, 55, 56, 57, 30].”
Reinforcement Learning with Self-Modifying Policies (1998, background): “There is no miraculous universal learning algorithm that will perform well in arbitrary environments [1, 2, 3].”
Nonlinear ICA through low-complexity autoencoders (1999, background): “For even more general, algorithmic methods reducing net complexity see [28].”
Evolving deep unsupervised convolutional networks for vision-based reinforcement learning (2014, background): “Most approaches to scaling neuroevolution to tasks that require large networks, such as those processing video input, have focused on indirect encodings where relatively small neural network descriptions are transformed via a complex mapping into networks of arbitrary size [4, 7, 10, 12, 14, 21].”
Evolving large-scale neural networks for vision-based TORCS (2013, background): “Therefore, recent efforts have focused on indirect encodings [1, 4, 5, 13] where relatively small genomes are transformed into networks of arbitrary size using a more complex mapping.”
Online Evolution of Deep Convolutional Network for Vision-Based Reinforcement Learning (2014, background): “Most approaches to scaling neuroevolution to tasks that require large networks, such as those processing video input, have focused on indirect encodings where relatively small neural network descriptions are transformed via a complex mapping into networks of arbitrary size [4, 7, 10,13,15,23].”
Drinking From a Firehose: Continual Learning With Web-Scale Natural Language (2020, methodology): “) The theory [74], [75] indicates that the first option will result in a high training loss and good generalization, while the second option will result in a low training loss and poor generalization.”
The Fastest and Shortest Algorithm for all Well-Defined Problems (2000, unknown): “On the other hand, Levin search has been successfully applied to solve rather difficult machine learning problems [14, 16], even though it suffers from a large multiplicative factor of similar origin.”
Deep neural networks are biased towards simple functions (2018, background): “The relation between generalization and complexity has been further investigated in [Schmidhuber, 1997, Dingle et al., 2018].”
Gödel Machines: Fully Self-referential Optimal Universal Self-improvers (2007, background): “Previous practical variants and extensions of universal search have been applied [36, 38, 50, 45] to offline program search tasks where the program inputs are fixed such that the same program always produces the same results.”
Feature Extraction Through LOCOCODE (1999, methodology): “Finally we would like to improve our understan- ding of the relationship of low-complexity codes, low-complexity art (see Schmidhuber, 1997b), and informal notions such as “beauty” and “good art.””