Instance-Conditioned GAN

By Arantxa Casanova, Marlene Careil, Jakob Verbeek, Michal Drozdzal, Adriana Romero-Soriano. 2021

164 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions of datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. In this paper, we take inspiration from kernel density estimation techniques and introduce a non-parametric approach to modeling distributions of complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors, and introduce a model, called instance-conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results on ImageNet and COCO-Stuff show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN can effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, we extend IC-GAN to the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet; while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available at https://github.com/facebookresearch/ic_gan.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, positive): “One closely related work to ours is the recent work on Instance-Conditioned GANs (IC-GAN) of Casanova et al. (2021).”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “Conditioning a GAN on pretrained features was also recently investigated by [Casanova et al. 2021].”
Flow Matching for Generative Modeling (2022, neutral): “Our FID is state-of-the-art with the exception of IC-GAN (Casanova et al., 2021) Sample paths.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, unknown): “However, current subject-driven text-to-image generation methods suffer from two key limitations: cost of personalization and identity blending for multiple subjects.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “1) Noises in Retrieval Results : Information retrieval cannot yield perfect results because information loss appears in representations generated by encoder models.”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, background): “Instance-Conditioned GAN [6] uses nearest neighbor images of a given reference for adversarial training to generate structurally diverse yet semantically relevant images.”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “In the evaluation phase, we follow common practice (Rombach et al. 2022; Anokhin et al. 2021; Casanova et al. 2021) to evaluate the performance on LSUN-bedand, CelebA-HQ, Landscape, where metrics are calculated between the 50k random generated images and the entired training set.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “Casanova et al. [11] propose an instance conditioned GAN that can generate variations of an instance, although it can struggle with unique subjects and does not preserve all subject details.”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Research in generative models has progressed rapidly in the last decade, with unconditional models capable of creating ever more realistic images (Dhariwal & Nichol, 2021; Casanova et al., 2021) and audio (Kong et al., 2020; Goel et al., 2022).”