Keeping the neural networks simple by minimizing the description length of the weights

By Geoffrey E. Hinton, Drew van Camp. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Deep AutoRegressive Networks (2013, methodology): “Training a DARN proceeds by minimising the total information stored for reconstruction of the original input, and as such follows the minimum description length principle (MDL; Rissanen, 1978).”
Towards Conceptual Compression (2016, methodology): “This second approach uses bits-back coding (Hinton & Van Camp, 1993).”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, background): “Regularization refers to the general approach of imposing a constraint on a model to make it harder to memorize the training data and therefore hopefully make it generalize better to unseen data [19].”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “As developed by Rissanen (1978) and further by Hinton & Van Camp (1993), holding accuracy constant, the best model is the one with the shortest MDL.”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, methodology): “Variational inference has been applied to these models, but with limited success (Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011).”
Sharp Minima Can Generalize For Deep Nets (2017, background): “We believe the reason for this contradiction stems from the Bayesian arguments about KL-divergence made to justify the generalization ability of ﬂat minima (Hinton & Van Camp, 1993).”
Variational Inference: A Review for Statisticians (2016, background): “In parallel, Hinton and Van Camp (1993) proposed a variational algorithm for a similar neural network model.”
Weight Agnostic Neural Networks (2019, background): “The weight parameters of a BNN [3, 4, 26, 43, 68, 78] are not ﬁxed values, but sampled from a distribution.”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Many have followed this approach in the past for standard NN models [26], [3], [19], [4].”
Variational Continual Learning (2017, methodology): “…to training neural networks, including extended Kalman ﬁltering (Singhal and Wu, 1989), Laplace approximation (MacKay, 1992), variational inference (Hinton and Van Camp, 1993; Barber and Bishop, 1998; Graves, 2011; Blundell et al., 2015; Gal and Ghahramani, 2016), sequential Monte Carlo (de…”