The Minimum Description Length Principle in Coding and Modeling

By Andrew R. Barron, Jorma Rissanen, Bin Yu. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Clustered Convolutional Kernels (2015, methodology): “Information theoretic methods, such as the minimum description length criterion, were also applied to the problem of structural optimization (Barron et al., 1998), resulting in improved performance in speech recognition (Shinoda and Watanabe, 2000) and as well as training algorithms for auto-encoder”
Exploiting compositionality to explore a large space of model structures (2012, background): “Minimum message length (Wal-lace, 2005), minimum description length (Barron et al., 1998), and Bayesian model comparison (MacKay, 1992) are frameworks which can, in principle, be used to compare very different generative models.”
User-friendly introduction to PAC-Bayes bounds (2021, background): “We refer the reader to [24, 87] for more recent presentations of this very fruitful approach.”
Motion Segmentation in the Presence of Outlying, Incomplete, or Corrupted Trajectories (2010, methodology): “coding length has been used effectively for data compression and model selection [1] as well as for segmentation”
Prediction, learning, and games (2006, background): “For quite exhaustive surveys see Barron, Rissanen, Yu [22], Grünwald [134], and Hansen and Yu [143].”
Face recognition with image sets using manifold density divergence (2005, methodology): “This can be done in a principled way with the Minimal Description Length (MDL) criterion [3].”
Sandwiching the marginal likelihood using bidirectional Monte Carlo (2015, background): “Third, it is closely related to description length (Barron et al., 1998), a compression-based criterion for model selection.”
Finding large average submatrices in high dimensional data (2009, background): “The penalty depends separately on k and l, and its combinatorial form suggests a connection with the Minimum Description Length Principle (MDL), following Rissanen (2004), Grunwald (2004), and Barron and Yu (1998). The MDL principle is a formalization of Occam’s Razor, in which the”
Combining Linear Regression Models (2005, background): “…and/or specific theoretical/empirical considerations, including the Akaike information criterion (AIC) (Akaike 1973), the Bayes information criterion (BIC) (Schwarz 1978), cross-validation (Allen 1974; Stone 1974), and Minimum Description Length (e.g., Rissanen 1984; Barron, Rissanen, and Yu 1998).”
A tutorial introduction to the minimum description length principle (2004, methodology): “We can get reasonable alternative universal models after all using any of the methods described in Section 2.7.2; see [Barron, Rissanen, and Yu 1998] and [Rissanen 2000] for details.”
Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It (2014, background): “In summary, we would urge both purists not to throw away the Bayesian baby with the misspecified bath water! Moreover, from a prequential (Dawid, 1984), learning theory (citations see below) and Minimum Description Length (MDL (Barron et al., 1998)) perspective, the extension from Bayes to SafeBayes”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, background): “The MDL criterion has undergone several fundamental revisions, such as the original crude MDL (Rissanen, 1978) and refined MDL (Rissanen, 1996; Barron et al., 1998).”
Sparse Modeling for Image and Vision Processing (2014, methodology): “Among a collection of plausible models, the simplest one is often preferred, and the number of underlying parameters is used as a criterion to perform model selection [Mallows, 1964, 1966, Akaike, 1973, Hocking, 1976, Barron et al., 1998, Rissanen, 1978, Schwarz, 1978, Tibshirani, 1996].”