The Description Length of Deep Learning models

By Léonard Blier, Yann Ollivier. 2018

116 citations recorded by the source corpus.

Deep learning models often have more parameters than observations, and still perform well. This is sometimes described as a paradox. In this work, we show experimentally that despite their huge number of parameters, deep neural networks can compress the data losslessly even when taking the cost of encoding the parameters into account. Such a compression viewpoint originally motivated the use of variational methods in neural networks. However, we show that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. Better encoding methods, imported from the Minimum Description Length (MDL) toolbox, yield much better compression values on deep networks.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Learning and Evaluating General Linguistic Intelligence (2019, methodology): “To directly quantify how quickly an existing model adapts to a new task we propose a new evaluation metric (§4) based on online (prequential) coding (Blier & Ollivier, 2018).”
True Few-Shot Learning with Language Models (2021, methodology): “Since ﬁnetuning is not possible without training data, we say that, in this case, A returns a uniform distribution over all labels, following prior work [e.g., 32].”
Weight Agnostic Neural Networks (2019, background): “Recent works offer a modern treatment [6] and application [61, 108] of these principles in the context of larger, deep neural network architectures.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “While variational methods achieve strong performance for moderately sized networks, they are empirically noted to be difﬁcult to train on larger architectures such as deep residual networks [22]; Blier and Ollivier [5] argue that the difﬁculty of training is explained by variational methods providin”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “In the NLP domain, Yogatama et al. (2019) introduced a new metric, based on an online (prequential) encoding (Blier and Ollivier, 2018), which measures the adoption rate of an existing model to a new task.”
Information-Theoretic Probing with Minimum Description Length (2020, neutral): “…is always correct (for accuracy higher is better, for codelength – lower). from different methods (Table 2) is rather surprising: this contrasts to Blier and Ollivier (2018), who experimented with images (MNIST, CIFAR-10) and argued that the variational code yields very poor compression bounds…”
A Bayesian Framework for Information-Theoretic Probing (2021, methodology): “In its online coding interpretation, it is rewritten as (Rissanen, 1978; Blier and Ollivier, 2018): Hθ(DN ) = ∑N n=1 Hθ(X | Dn−1)—where the cross-entropy of each element X in DN is computed incrementally because the parameter θ (which would make them independent) is unknown.”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, methodology): “These methods are all compatible with RDA, and here, we use online (or prequential) coding (Rissanen, 1984; Dawid, 1984), an effective method for estimating MDL when used with deep learning (Blier and Ollivier, 2018).”
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights (2020, background): “Another way to measure it is through the intrinsic dimension of an objective landscape (Li et al., 2018; Blier & Ollivier, 2018).”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, background): “Recently MDL has been ported to DNNs [8] focusing on variational methods.”
Pareto Probing: Trading-Off Accuracy and Complexity (2020, methodology): “A recent proposal by Voita and Titov (2020) suggests that minimum description length (MDL; Ris-sanen, 1978) is a useful approach to the problem of balancing performance and complexity.”