Stochastic Complexity and Modeling

By Jorma Rissanen. 1986

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “Somemethods explicitly take into account program runtime (Allender, 1992; Schmidhuber, 1997, 2002; Watanabe, 1992); many consider only programs with constant runtime, written in non-universal programming languages (e.g., Hinton & van Camp, 1993; Rissanen, 1986).”
Automated Curriculum Learning for Neural Networks (2017, methodology): “These gains are underpinned by the Minimum Description Length (MDL) principle (Rissanen, 1986; Gr ¨ unwald, 2007): in order to best generalise from a particular dataset, one should minimise the number of bits required to describe the model parameters plus the number of bits required for the model…”
Bayesian Compression for Deep Learning (2017, background): “It relates to compression directly in that it defines the best hypothesis to be the one that communicates the sum of the model (complexity cost L) and the data misfit (error cost L) with the minimum number of bits [59, 60].”
Soft Weight-Sharing for Neural Network Compression (2017, background): “More speciﬁcally, it minimizes the cost to describe the model (complexity cost L C ) and the misﬁt between model and data (error cost L E ) (Rissanen, 1978; 1986).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “Instead, to avoid the erroneous “discovery” of “regular patterns” in irregular noise, we use AIT’s sound way of dealing with overfitting [263, 130, 289, 207, 147, 84], and measure M ’s compression performance by the number of bits required to specify M , plus the bits needed to encode the observed d”
View independent human body pose estimation from a single perspective image (2004, unknown): “Thiseffectivelyaccountedfor variationsin theassumedand actuallimb lengths.”
Algorithmic Complexity of Financial Motions (2014, unknown): “…of empirical works where concepts from information theory, such as Shannon's entropy (Shannon, 1948), Kolmogorov complexity (Kolmogorov, 1965) and stochastic complexity (Rissanen, 1986) are used to highlight the similarities and dissimilarities between random strings and real world financial data.”
MUTUAL INFORMATION, METRIC ENTROPY AND CUMULATIVE RELATIVE ENTROPY RISK (1997, background): “[50] J. Rissanen.”
Deterministic annealing for clustering, compression, classification, regression, and related optimization problems (1998, background): “In one basic approach, penalty terms are added to the training cost, either to directly favor the formation of a small model [1], [85], or to do so indirectly via regularization/smoothness constraints or other costs which measure overspecialization.”
Keeping the neural networks simple by minimizing the description length of the weights (1993, methodology): “The Minimum Description Length Principle ( Rissanen, 1986 ) asserts that the best model of some data is the one that minimizes the combhmd cost of describing the model and describing the misfit between the model and the data.”
Prediction, learning, and games (2006, unknown): “This notion generalizes Rissanen’s stochastic complexity [240, 242].”
The Graphical Horseshoe Estimator for Inverse Covariance Matrices (2017, background): “Theorem 1 of Rissanen (1986) gives an asymptotic lower bound on the Kullback–Leibler divergence D(p 0 ||p̂n), which is (1/2 − ) k log n for all > 0, where k is the dimension of the parameter vector.”
Asymptotic statistical theory of overtraining and cross-validation (1997, methodology): “For example, model selection methods (e.g., [23], [20], [26] and many others), regularization ([25] and others), and early stopping ([16], [15], [30], [11], [4] and others) or structural risk minimization (SRM, cf. [32]) can be applied.”
Adaptive computation and machine learning (1998, background): “, the minimum description length principle (Rissanen, 1986) and various statistical selection criteria (Foster and George, 1994)); and holdout testing (e.”