Unsupervised multi-modal Styled Content Generation

By Omry Sendik, Dani Lischinski, Daniel Cohen-Or. 2020

4 citations recorded by the source corpus.

The emergence of deep generative models has recently enabled the automatic generation of massive amounts of graphical content, both in 2D and in 3D. Generative Adversarial Networks (GANs) and style control mechanisms, such as Adaptive Instance Normalization (AdaIN), have proved particularly effective in this context, culminating in the state-of-the-art StyleGAN architecture. While such models are able to learn diverse distributions, provided a sufficiently large training set, they are not well-suited for scenarios where the distribution of the training data exhibits a multi-modal behavior. In such cases, reshaping a uniform or normal distribution over the latent space into a complex multi-modal distribution in the data domain is challenging, and the generator might fail to sample the target distribution well. Furthermore, existing unsupervised generative models are not able to control the mode of the generated samples independently of the other visual attributes, despite the fact that they are typically disentangled in the training data. In this paper, we introduce UMMGAN, a novel architecture designed to better model multi-modal distributions, in an unsupervised fashion. Building upon the StyleGAN architecture, our network learns multiple modes, in a completely unsupervised manner, and combines them using a set of learned weights. We demonstrate that this approach is capable of effectively approximating a complex distribution as a superposition of multiple simple ones. We further show that UMMGAN effectively disentangles between modes and style, thereby providing an independent degree of control over the generated content.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Samuli Laine
2. Timo Aila
3. Janne Hellsten
4. Miika Aittala
5. Tero Karras
6. Jaakko Lehtinen
7. Guo-Jun Qi
8. Kai Han

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Antonio Torralba
4. Jun-Yan Zhu
5. Eli Shechtman
6. Dacheng Tao
7. Richard Zhang
8. Chen Change Loy

In Natural Language Processing

1. Timo Aila
2. Samuli Laine

In Multimodal Learning

1. Timo Aila
2. Miika Aittala
3. Tero Karras

Citations in the corpus

• Training Generative Adversarial Networks with Limited Data (2020, background): “As future work, it would be worthwhile to search for the most effective set of augmentations, and to see if recently published techniques, such as the U-net discriminator [38] or multi-modal generator [39], could also help with limited data.”