ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching

By Chunyuan Li, Hao Liu, Changyou Chen, Yunchen Pu, Liqun Chen, Ricardo Henao, Lawrence Carin. 2017

227 citations recorded by the source corpus.

We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional entropy, we propose both adversarial and non-adversarial approaches to learn desirable matched joint distributions for unsupervised and supervised tasks. We unify a broad family of adversarial models as joint distribution matching problems. Our approach stabilizes learning of unsupervised bidirectional adversarial learning methods. Further, we introduce an extension for semi-supervised learning tasks. Theoretical results are validated in synthetic data and real-world applications.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Towards Language-Free Training for Text-to-Image Generation (2021, background): “GANs [11] have inspired many advances in image synthesis [18, 20, 23, 28, 32].”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Therefore, our argument provides an explanation as to why we observe better sample quality results from GANs and diffusion models than VAEs and NFs.”
GAN Inversion: A Survey (2021, background): “[182] describe the non-identifiability issue of ALI and provide a unified framework for recently proposed GAN”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, neutral): “This is conceptually related to several previous GAN models in the image domain that designed for joint distribution matching [13, 25, 35, 36, 37].”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “Style-augmented Cycle Consistency Joint distribution matching can be realized via cycle consistency constraint [8], assuming deterministic translation models and matched marginals [43, 67, 68].”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, neutral): “For example [16], [17] replace the KL divergence with the Jensen-Shannon divergence which enables the data and latent codes to be treated in a symmetric manner.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “odels which would only train generator and inference models within a single optimization function [36] or learn an optimal coupling between generator and inference model by using adversarial learning [11,15,16,36,37,42,43,61]. The proposed algorithm contains two independent optimization paths, namel”
Learning Implicit Fields for Generative Shape Modeling (2018, methodology): “We also report the inception score for MNIST (IS) [26] and the log-likelihood produced by Parzen-window estimate (PWE) [4, 5, 15].”
Generative Models from the perspective of Continual Learning (2018, background): “ally the activation should be taken from a given layer of a given Inception-v3 instance, however this setting can be adapted with another classiﬁer in order to compare a set of models with each other [28, 14]. A different approach is to use labeled generated samples from a generator G (GAN or VAE) t”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Li et al. [80] point out a non-identiﬁability issue inherent with the distribution matching problem underlying ALI/BiGAN, and propose to penalize the entropy of the reconstruction conditionally on the code.”
Mutual Information Neural Estimation (2018, background): “Reconstructions are one desirable property of a model that does both inference and generation, but in practice ALI can lack ﬁdelity (i.e., reconstructs less faithfully than desired, see Li et al., 2017; Ulyanov et al., 2017; Belghazi et al., 2018).”
Distribution Matching in Variational Inference (2018, methodology): “A previously proposed solution explores joint distribution matching (Donahue et al., 2017, Dumoulin et al., 2017, Li et al., 2017, Srivastava et al., 2017).”
Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model (2018, methodology): “Table 4 shows the IS and FID scores of our model compared with VAE [15], DCGAN [20], WGAN [22], CoopNet [25], CEGAN [24], ALI [26], ALICE [28].”
Adversarial Symmetric Variational Autoencoder (2017, background): “However, non-identifiability issues are raised in [32].”
Point Cloud GAN (2018, background): “Most of them are the dual form of f -divergence (Goodfellow et al., 2014; Mao et al., 2017; Nowozin et al., 2016), integral probability metrics (IPMs) (Zhao et al., 2016; Li et al., 2017a; Arjovsky et al., 2017; Gulrajani et al., 2017) or IPM extensions (Mroueh and Sercu, 2017; Mroueh et al., 2017).”
MINE: Mutual Information Neural Estimation (2018, methodology): “However, in practice ALI guarantees neither due to identifiability issues (Li et al., 2017).”
Learning to Sketch with Shortcut Cycle Consistency (2018, unknown): “Cycle consistency is further added as a way to transitively regularize structured data, which greatly alleviates non-identiﬁability issues [20].”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “The ideas in (Li et al., 2017b) are similar to ours, but the motivation and underlying theory are different from that developed here.”