Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models

By Jesse Engel, Matthew D. Hoffman, Adam Roberts. 2017

151 citations recorded by the source corpus.

Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions. Conditional generation enables interactive control, but creating new controls often requires expensive retraining. In this paper, we develop a method to condition generation without retraining the model. By post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions. Combining attribute constraints with a universal "realism" constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder. Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make the minimal adjustment in latent space to modify the attributes of an image. Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. Code with dedicated cloud instance has been made publicly available (this https URL).

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Prompting & Reasoning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Prompting & Reasoning

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, methodology): “Furthermore, classifiers [54, 55, 56], adversarial methods [57], and other techniques [58, 59] have been used to define prior distributions implicitly.”
Symbolic Music Generation with Diffusion Models (2021, methodology): “Unlike many existing deep generative models, DDPMs sample through an iterative reﬁnement process inspired by Langevin dynamics [3], which enables post-hoc conditioning of models trained unconditionally [4–7] for creative applications.”
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (2017, background): “A similar idea to using such concept vectors in latent dimension in the context of generative model has also been explored (Engel et al., 2017).”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “Additionally, future work could explore the utility of controlling model generation using labelled “undesirable” examples (Zhang et al., 2020; Engel et al., 2017), or human-in-the-loop learning methods (Wang et al.”
Structured Disentangled Representations (2018, methodology): “In settings where we are not primarily interested in inducing disentangled representations, the β -VAE objective has also been used with β 1 in order to improve the quality of reconstructions [Alemi et al., 2016, Engel et al., 2017, Liang et al., 2018].”
Conditioning by adaptive sampling for robust design (2019, methodology): “This approach in turn has a resemblance to Engel et al. (2018), wherein the goal is image generation, and a GAN objective is placed on top of a VAE in order to learn latent constraints. Because this latter approach was designed for (real-valued) images and for classification labels, we compare only ”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, methodology): “Our interest in deep latent variable models primarily comes from their increasing use in creative applications of machine learning (Carter & Nielsen, 2017; Ha & Eck, 2018; Engel et al., 2017a).”
Guided Variational Autoencoder for Disentanglement Learning (2020, background): “There are also approaches [14, 5] in which latent variables constraints are added, but they have different formulations and objectives than Guided-VAE.”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “Adversarial loss has been used for training implicit encoders in VAEs (Makhzani et al., 2015; Mescheder et al., 2017; Engel et al., 2018).”
Latent Translation: Crossing Modalities by Bridging Generative Models (2019, methodology): “For MNIST and Fashion MNIST, we pretrain VAEs with MLP encoders and decoders following Engel et al. (2018).”
Towards Text Generation with Adversarially Learned Neural Outlines (2018, methodology): “This aligns with observations made in [14], where VAEs trained with a variance of ∼ 0.”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “, 2017), or training a posthoc prior (Engel et al., 2017; Ghosh et al., 2019).”
Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Text (2020, background): “(Engel, Hoffman, and Roberts 2017) enforces behavioral latent constraints as part of post-hoc learning on pre-trained generative models, whereas (Dathathri et al. 2020) incorporates attribute discriminators to steer the text generations.”
Controlling generative models with continuous factors of variations (2020, background): “Similarly regarding VAE, Engel et al. (2018) identiﬁed that they suffer from a trade-off between reconstruction accuracy and sample plausibility and proposed to identify regions of the latent space that correspond to plausible samples to improve reconstruction accuracy.”
Diffusion models as plug-and-play priors (2022, background): “In this paper, we show that DDPMs trained on image data can be directly used as priors in systems that involve other differentiable constraints.”
Point Cloud GAN (2018, background): “The other variant of AAE is training the other generative model to learn the distribution of the encoded data instead of enforcing it to be similar to a known distribution (Engel et al., 2017; Kim et al., 2017).”
Encoding Musical Style with Transformer Autoencoders (2019, background): “There has been signiﬁcant progress in generative modeling, particularly with respect to creative applications such as art and music (Oord et al., 2016; Engel et al., 2017b; Ha & Eck, 2017; Huang et al., 2019a; Payne, 2019).”
A de novo molecular generation method using latent vector based generative adversarial network (2019, methodology): “For the later application, approaches that have utilized multiple discriminators have been used to combine conditional VAEs and conditional GANs to enforce constraints on the latent space [51] and thus increase the realism of the images.”