MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation

By Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, Yong Jae Lee. 2019

84 citations recorded by the source corpus.

We present MixNMatch, a conditional generative model that learns to disentangle and encode background, object pose, shape, and texture from real images with minimal supervision, for mix-and-match image generation. We build upon FineGAN, an unconditional generative model, to learn the desired disentanglement and image generator, and leverage adversarial joint image-code distribution matching to learn the latent factor encoders. MixNMatch requires bounding boxes during training to model background, but requires no other supervision. Through extensive experiments, we demonstrate MixNMatch's ability to accurately disentangle, encode, and combine multiple factors for mix-and-match image generation, including sketch2color, cartoon2img, and img2gif applications. Our code/models/demo can be found at https://github.com/Yuheng-Li/MixNMatch

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

Counterfactual Generative Networks (2021, background): “Closely related to our work is (Li et al., 2020), which aims to disentangle background, shape, pose, and texture, using object bounding boxes for supervision.”
The Creation and Detection of Deepfakes (2020, background): “Similarly, the authors of [85] proposed a method disentangled identity transfer.”
Towards Universal Fake Image Detectors that Generalize Across Generative Models (2023, background): “The input for generating such images could be random noise [28, 29], categorical class information [7], text prompts [31,36,42,46], or could even by a collection of images [32].”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “A large number of work try to disentangle object shape/structure from appearance [41,27,39,7,51].”
Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs (2020, methodology): “Thus, we must disentangle [14,33,28] v and z using a mapping t(·|z) : z̄ 7→ v = t(z̄|z) which, depending on z, extracts v from z̄.”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “MixNMatch [22] disentangled multiple factors including background, pose, shape and texture by decomposing image synthesis into multiple steps.”