A Lagrangian Perspective on Latent Variable Generative Models

By Shengjia Zhao, Jiaming Song, Stefano Ermon. 2018

20 citations recorded by the source corpus.

A large number of objectives have been proposed to train latent variable generative models. We show that many of them are Lagrangian dual functions of the same primal optimization problem. The primal problem optimizes the mutual information between latent and visible variables, subject to the constraints of accurately modeling the data distribution and performing correct amortized inference. By choosing to maximize or minimize mutual information, and choosing different Lagrange multipliers, we obtain different objectives including InfoGAN, ALI/BiGAN, ALICE, CycleGAN, beta-VAE, adversarial autoencoders, AVB, AS-VAE and InfoVAE. Based on this observation, we provide an exhaustive characterization of the statistical and computational trade-offs made by all the training objectives in this class of Lagrangian duals. Next, we propose a dual optimization method where we optimize model parameters as well as the Lagrange multipliers. This method achieves Pareto optimal solutions in terms of optimizing information and satisfying the constraints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Stefano Ermon
2. Jiaming Song
3. Chenlin Meng
4. Shengjia Zhao
5. Sergey Levine
6. Aditya Grover
7. Fabio Viola
8. Danilo Jimenez Rezende

In Computer Vision and Pattern Recognition

1. Stefano Ermon
2. Jiaming Song
3. Luc Van Gool
4. Ming-Hsuan Yang
5. Chenlin Meng
6. Trevor Darrell
7. Dacheng Tao
8. Jitendra Malik

In General Social Sciences

1. Stefano Ermon
2. Jiaming Song
3. Aditya Grover

In Generative Models

1. Stefano Ermon
2. Jiaming Song
3. Chenlin Meng
4. Arash Vahdat
5. Fabio Viola
6. Shengjia Zhao
7. Danilo Jimenez Rezende
8. Karsten Kreis

In Probabilistic & Bayesian Methods

1. Stefano Ermon
2. Chenlin Meng
3. Jiaming Song
4. Shengjia Zhao
5. Aditya Grover
6. Abhishek Sinha
7. Danilo Jimenez Rezende
8. Fabio Viola

Citations in the corpus

• D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
• Learning Controllable Fair Representations (2018, background): “Lagrange multipliers during training (Zhao et al., 2018), thereby also learning the trade-off between expressiveness and fairness.”
• Generalized ELBO with Constrained Optimization , GECO (2018, background): “At a high-level, GECO is similar to information constraints studied in [20, 29, 30, 31, 32].”
• InfoVAE: Balancing Learning and Inference in Variational Autoencoders (2019, background): “Relation to Information Theoretical Analysis: Several other lines of work analyze mutual information in latent variable generative models from different perspectives, including (Zhao, Song, and Ermon 2018; Alemi et al. 2017).”