Modeling By Shortest Data Description*

By Jorma Rissanen. 1978

Subfields: Statistical Learning Theory, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

In Information Theory

Citations in the corpus

True Few-Shot Learning with Language Models (2021, methodology): “We may also form train-validation splits in a different manner than CV, drawing inspiration from work on the Minimum Description Length (MDL) principle [28].”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “As developed by Rissanen (1978) and further by Hinton & Van Camp (1993), holding accuracy constant, the best model is the one with the shortest MDL.”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, neutral): “Voita and Titov (2020) follow this direction, using the concept of minimum description length (MDL, Rissanen, 1978) to quantify the total information needed to transmit both the probing model and the labels it predicts.”
Weight Agnostic Neural Networks (2019, methodology): “The Minimal Description Length (MDL) [34, 91, 92] is a formalization of Occam’s razor, in which a good model is one that is best at compressing its data, including the cost of describing of the model itself.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “For learning disentangled representations, we employ the Minimum Description Length (MDL) principle [7, 57] and replace the second term from equation (20) by γ|DKL[qς(z|x)||p(z)]−C|, where γ and C are a multiplicative and a linear constant used for controling the degree of disentanglement.”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, methodology): “These approaches can broadly be seen as efficient implementations of the Minimum Description Length (MDL) principle for unsupervised learning [44, 18].”
Bayesian Compression for Deep Learning (2017, background): “It relates to compression directly in that it defines the best hypothesis to be the one that communicates the sum of the model (complexity cost L) and the data misfit (error cost L) with the minimum number of bits [59, 60].”
Inductive Logic Programming: Theory and Methods (1994, background): “The principle of choosing the theory which minimises I(T jE) is known as Rissanen's minimum description length principle (MDL)[119]6.”
Soft Weight-Sharing for Neural Network Compression (2017, methodology): “More specifically, it minimizes the cost to describe the model (complexity cost LC) and the misfit between model and data (error cost LE) (Rissanen, 1978; 1986).”
Community detection in graphs (2009, methodology): “The Minimum Description Length (MDL) principle (Grünwald et al., 2005; Rissanen, 1978) provides a solution to this problem, which amounts to the minimization of a function given by H(X|Y ) plus a function of the number n of vertices, m of edges and q of clusters.”
Reversible Jump Markov Chain Monte Carlo Strategies for Bayesian Model Selection in Autoregressive Processes (2004, methodology): “BIC and MDL yield the same rule, which is asymptotically consistent.”
Consistent Model Selection by an Automatic "Gets" Approach (2003, methodology): “The Schwarz (1978) information criterion,SIC (independently derived by Rissanen, 1978, using coding theory) selects from the set ofn candidate variables in (1) the model withk regressors and parameter estimates̃δi which minimizes: SICk = ln σ̃ 2 k + c k ln T T .”
Reliable Intrinsic Connectivity Networks: Test-Retest Evaluation Using ICA and Dual Regression Approach (2009, unknown): “For each individual resting state fMRI dataset, we applied five algorithms to estimate the number of components automatically: Akaike information criterion (Akaike, 1973), Bayesian information criterion (Schwarz, 1978), minimum description length (Rissanen, 1978), probabilistic PCA (Minka, 2001) and”
V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure (2007, methodology): “While this is an application of the minimum description length (MDL) principle (Rissanen, 1978; Rissanen, 1989), it does not provide an intuitive manner for assessing our two competing criteria of homogeneity and completeness.”