Information Theory, Inference, and Learning Algorithms

By David John Cameron MacKay. 2004

10,960 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Electrical and Electronic Engineering, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep AutoRegressive Networks (2013, background): “The idea is that the representation has a more concise code than the original datum x and so can be compressed effectively: for example, by arithmetic coding (MacKay, 2003).”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “The gradient of the KL-divergence can be written as (MacKay, 2003):”
Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, background): “Sampling from such distributions is a critical task for learning and inference (MacKay, 2003), however it is an extremely hard problem in general.”
Variational Diffusion Models (2021, background): “Likelihood-based generative modeling is a central task in machine learning that is the basis for a wide range of applications ranging from speech synthesis [Oord et al., 2016], to translation [Sutskever et al., 2014], to compression [MacKay, 2003], to many others.”
How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? (2015, background): “The differences in behaviour between KL [ Q (cid:107) P ] and KL [ P (cid:107) Q ] are well understood and exploited for example in the context of approximate Bayesian inference (Lacoste-Julien et al., 2011; MacKay, 2003; Minka, 2001).”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, methodology): “Algorithm 3 illustrates the entire process. add digit from buffer 7: i = i ÷ n shorten information buffer 8: return updated buffer i , updated value c We could also have used an arithmetic coding scheme for our information buffer (MacKay, 2003, Chapter 6).”
From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews (2013, methodology): “We settle for a local optimum, and optimize the parameters Θ and E us-ing coordinate ascent [27].”
Deep Variational Information Bottleneck (2017, background): “5 A dozen samples wasn’t chosen for any particular reason, except the old addage that a dozen samples are sufficient, as mirrored in David MacKay’s book (MacKay, 2003).”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, neutral): “[42] MacKay, D.”
Fastfood: Approximate Kernel Expansions in Loglinear Time (2014, neutral): “See e.g. (Williams, 1998; Neal, 1994; MacKay, 2003) for details.”
Direct-coupling analysis of residue coevolution captures native contacts across many protein families (2011, methodology): “5 contains more free parameters than there are independent conditions in [4], which allows one to change couplings and fields together without changing the sum in the exponent.”
Progress & Compress: A scalable framework for continual learning (2018, background): “Unfortunately, the exact posteriors needed are intractable, and are replaced by Laplace’s approximation (MacKay, 2003).”
Posterior Meta-Replay for Continual Learning (2021, background): “Predictions are made by marginalizing over models (for an introduction see MacKay [47]).”
A contextual-bandit approach to personalized news article recommendation (2010, background): “Furthermore, in information theory [19], the diff erential entropy ofp(θa) is defined as− 1 2 ln((2π) detAa).”