Ensembling with Deep Generative Views

By Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, Richard Zhang. 2021

74 citations recorded by the source corpus.

Recent generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose, simply by learning from unlabeled image collections. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. Using a pre-trained generator, we first find the latent code corresponding to a given real input image. Applying perturbations to the code creates natural variations of the image, which can then be ensembled together at test-time. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars. Critically, we find that several design decisions are required towards making this process work; the perturbation procedure, weighting between the augmentations and original image, and training the classifier on synthesized images can all impact the result. Currently, we find that while test-time ensembling with GAN-based augmentations can offer some small improvements, the remaining bottlenecks are the efficiency and accuracy of the GAN reconstructions, coupled with classifier sensitivities to artifacts in GAN-generated images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Exploring new editing methods and dataset generation [Chai et al. 2021; Li et al. 2022] using StyleGAN-XL are exciting future avenues.”
LARGE: Latent-Based Regression through GAN Semantics (2021, methodology): “While these works cover a diverse collection of settings and approaches, they all used GANs in order to eventually generate more data for training .”
Diffusion Models for Adversarial Purification (2022, methodology): “We use the codebase: https://github.com/chail/gan-ensembling corresponding to the idea of (Chai et al., 2021) that uses the w+ latent code from the encoder as the initial point for the optimization with n = 500 iterations.”
GAN-Supervised Dense Visual Alignment (2021, methodology): “We do not rely on hand-crafted pixel space augmentations [12, 35], human-labeled data [76, 86, 87] or post-processing of GANgenerated datasets using domain knowledge [10, 55, 79, 86].”