The Minimum Description Length Principle (Adaptive Computation and Machine Learning)

By Peter D. Grünwald. 2007

894 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistics and Probability, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistics and Probability

In Scaling & Learning Theory

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, methodology): “Its size can therefore be considered a measure of the minimum description length [13] of the data using a two part coding scheme.”
Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks (2014, background): “…(empirical) Bayesian marginal likelihoods can be interpreted as measuring likelihood in combination with a complexity penalty, Minimum Message Length (MML) techniques can be used to construct scores that incorporate a trade-off between model fit (likelihood) and model complexity (Grünwald, 2007).”
Automated Curriculum Learning for Neural Networks (2017, methodology): “ turn to a novel set of gains that instead measure the rate at which the network’s complexity increases. These gains are underpinned by the Minimum Description Length (MDL) principle (Rissanen, 1986; Grunwald, 2007): in order to best generalise from a¨ particular dataset, one should minimise the num”
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data (2017, methodology): “Hochreiter and Schmidhuber discuss sharp versus flat minima using the language of minimum description length (MDL; [Ris83; Grü07]).”
Rényi Divergence and Kullback-Leibler Divergence (2012, background): “As is well known [3], [58], the distribution that minimizes the worst-case regret is uniquely given by the normalized maximum likelihood or Shtarkov distribution S(x) = supθ Pθ(x) ∑ x supθ Pθ(x) , (66)”
Suboptimal behavior of Bayes and MDL in classification under misspecification (2004, background): “3 in Chapter 15 of [Grünwald, 2007].”
Origo: causal inference by compression (2016, methodology): “Kolmogorov complexity is not computable, however, and hence we derive a practical, computable version based on the Minimum Description Length (MDL) principle [8], [9].”
A Minimum Relative Entropy Principle for Learning and Acting (2008, background): “For example, one could think of partitioning a continuous space of operation modes into “essentially different” regions where representative operation modes subsume their neighborhoods (Grünwald, 2007).”
The power of quantum neural networks (2020, methodology): “In computational learning theory, it is used to measure complexity according to the principle of minimum description length [42].”
Catching up faster by switching sooner: a predictive approach to adaptive estimation with an application to the AIC–BIC dilemma (2012, methodology): “…not only the standard Kullback–Leibler risk but also the cumulative risk R.pÅ, p, n/= n∑ i=1 r.pÅ, p, i/: The cumulative risk is equal to the information theoretic redundancy, i.e. the Kullback–Leibler divergence on n outcomes (see for example Barron (1998a) or Grünwald (2007), chapter 15).”
The Challenge of Machine Learning in Space Weather: Nowcasting and Forecasting (2019, background): “, 1978), the Akaike Information Criterion (AIC) (Akaike, 1998), and the Minimum Description Length (Grünwald, 2007).”