GAN-Supervised Dense Visual Alignment

By William S. Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A. Efros, Eli Shechtman. 2021

78 citations recorded by the source corpus.

We propose GAN-Supervised Learning, a framework for learning discriminative models and their GAN-generated training data jointly end-to-end. We apply our framework to the dense visual alignment problem. Inspired by the classic Congealing method, our GAN gealing algorithm trains a Spatial Transformer to map random samples from a GAN trained on unaligned data to a common, jointly-learned target mode. We show results on eight datasets, all of which demonstrate our method successfully aligns complex data and discovers dense correspondences. GANgealing significantly outperforms past self-supervised correspondence algorithms and performs on-par with (and sometimes exceeds) state-of-the-art supervised correspondence algorithms on several datasets-without making use of any correspondence supervision or data augmentation and despite being trained exclusively on GAN-generated data. For precise correspondence, we improve upon state-of-the-art supervised methods by as much as 3 ×. We show applications of our method for augmented reality, image editing and automated pre-processing of image datasets for downstream GAN training.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “As generative models continue to improve, there is growing interest in their use as a source of cheap and plentiful training data for downstream tasks [33, 47, 50, 58, 65, 66].”
Emergent Correspondence from Image Diffusion (2023, background): “In particular, recent work has also found that the internal representation of Generative Adversarial Networks (GAN) [24] could be used for identifying visual correspondence [94, 63, 58] within certain image categories.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “To evaluate the correspondence accuracy, we use the standard Percentage of Correct Keypoints (PCK) [67] metric with a threshold of κ·max(h,w) where κ is a positive integer, and (h,w) denotes the dimensions of the bounding box of an instance in SPair-71k, and the dimensions of the images in PF-Pascal”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, unknown): “49 DiT-XL/2 (Peebles & Xie, 2022) [42] 9.”
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, methodology): “GANs [29] have been used widely for dataset generation on tasks such as visual alignment [62], face manipulation [82], and adversarial training for image synthesis [74].”