The IM algorithm: a variational approach to Information Maximization

By David Barber, Felix V. Agakov. 2003

535 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

In Information Theory

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “This technique of lower bounding mutual information is known as Variational Information Maximization [26].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “In addition, we can recover the recently proposed InfoNCE [55] bound on mutual information by using SNIS as the variational distribution in the classic variational bound by Barber and Agakov [4] (see Appendix C for details).”
On Mutual Information Maximization for Representation Learning (2019, methodology): “Given the fundamental limitations of MI estimation (McAllester and Statos, 2018), recent work has focused on deriving lower bounds on MI (Barber and Agakov, 2003; Belghazi et al., 2018; Poole et al., 2019).”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “To provide a principled approach to maximizing MI, we adopt variational information maximization [9, 10].”
Deep Variational Information Bottleneck (2017, background): “Variational bounds on mutual information have previously been explored in Agakov (2004), though not in conjunction with the information bottleneck objective.”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “This technique is known as the variational information maximization [1].”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “InfoGAN learns to disentangle zf by maximizing the mutual information between zf and the observed data x, using the following objective function: V InfoGAN(D,G) = V GAN(D,G)− λI (zf, G(z)) (2) ≤ V GAN(D,G)− λV Info(G,Q) (3) where I(·, ·) denotes the mutual information; λ > 0; and V Info(G,Q) is a lo”
Novel Class Discovery without Forgetting (2022, methodology): “Numerically computing exact mutual information is intractable, and hence we resort to a variational approximation q(l|u) to true distribution p(l|u) [2,3] as follows:”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “Furthermore, det ( J xθ ) denotes the determinant of the Jacobian of F θ ( x ) and The underlying principles of the nuisance classiﬁcation loss L nCE can be understood using a variational lower bound on mutual information from (Barber & Agakov, 2003).”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Jensen’s Inequality tells us that replacing p(z | s) with qφ(z | s) gives us a variational lower bound G(θ, φ) on our objective F(θ) (see (Agakov, 2004) for a detailed derivation): F(θ) = H[A | S,Z]−H[Z | S] +H[Z] = H[A | S,Z] + Ez∼p(z),s∼π(z)[log p(z | s)]− Ez∼p(z)[log p(z)] ≥ H[A | S,Z] + Ez∼p(z),”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, methodology): “The MI can be made more tractable by deriving a lower bound to it and maximising this instead — here we present the bound derived by Barber & Agakov [1].”
On Variational Bounds of Mutual Information (2019, methodology): “As shown in Barber & Agakov (2003), this yields a lower bound on MI due to the non-negativity of the KL divergence: where h ( X ) is the differential entropy of X .”
Mutual Information Neural Estimation (2018, background): “In order to overcome the intractability of I ( X ; Z ) in the continuous setting, Alemi et al. (2016); Kolchinsky et al. (2017); Chalk et al. (2016) exploit the variational bound of Barber & Agakov (2003) to approximate the conditional entropy in I ( X ; Z ) .”
PixelGAN Autoencoders (2017, background): “Indeed, it can be shown that minimizing the reconstruction term maximizes a variational lower bound on I ( z ; x ) [17, 18].”
Gradient Estimators for Implicit Models (2017, methodology): “In the context of GANs, the Jensen-Shannon divergence is approximated by a variational lower-bound represented by a discriminator [3, 10].”