Learning Representations by Maximizing Mutual Information in Variational Autoencoders

By Ali Lotfi-Rezaabad, Sriram Vishwanath. 2019

42 citations recorded by the source corpus.

Variational autoencoders (VAE) have ushered in an new era of unsupervised learning methods for complex distributions. Although these techniques are elegant in their approach, they are typically not useful for representation learning. In this work, we propose a simple yet powerful class of VAEs that simultaneously result in meaningful learned representations. Our solution is to combine traditional VAEs with mutual information maximization, with the goal to enhance amortized inference in VAEs using Information Theoretic techniques. We call this approach InfoMax-VAE, and such an approach can significantly boost the quality of learned high-level representations. We realize this through explicit maximization of information measures associated with the representation. Using extensive experiments on varied datasets and setups, we show that InfoMax-VAE outperforms contemporary popular approaches, including Info- VAE and β-VAE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Behavior-Driven Synthesis of Human Dynamics (2021, background): “This becomes evident as the regularization DKL(qφ(zβ |x, xt)||p(zβ)) minimizes an upper bound on the mutual information Iqφ(x; zβ) [47], thus reducing the information captured in zβ .”