MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking

By Rushil Anirudh, Jayaraman J. Thiagarajan, Bhavya Kailkhura, Timo Bremer. 2019

47 citations recorded by the source corpus.

In the past few years, Generative Adversarial Networks (GANs) have dramatically advanced our ability to represent and parameterize high-dimensional, non-linear image manifolds. As a result, they have been widely adopted across a variety of applications, ranging from challenging inverse problems like image completion, to problems such as anomaly detection and adversarial defense. A recurring theme in many of these applications is the notion of projecting an image observation onto the manifold that is inferred by the generator. In this context, Projected Gradient Descent (PGD) has been the most popular approach, which essentially optimizes for a latent vector that minimizes the discrepancy between a generated image and the given observation. However, PGD is a brittle optimization technique that fails to identify the right projection (or latent vector) when the observation is corrupted, or perturbed even by a small amount. Such corruptions are common in the real world, for example images in the wild come with unknown crops, rotations, missing pixels, or other kinds of non-linear distributional shifts which break current encoding methods, rendering downstream applications unusable. To address this, we propose corruption mimicking—a new robust projection technique, that utilizes a surrogate network to approximate the unknown corruption directly at test time, without the need for additional supervision or data augmentation. The proposed method is significantly more robust than PGD and other competing methods under a wide variety of corruptions, thereby enabling a more effective use of GANs in real-world applications. More importantly, we show that our approach produces state-of-the-art performance in several GAN-based applications—anomaly detection, domain adaptation, and adversarial defense, that benefit from an accurate projection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “…al., 2020; Wu et al., 2021), designing better encoders (Richardson et al., 2021; Tov et al., 2021), modeling image corruption and transformations (Anirudh et al., 2020; Huh et al., 2020), and discovering meaningful latent directions (Shen et al., 2020; Goetschalckx et al., 2019; Jahanian et al.,…”
Transforming and Projecting Images into Class-conditional Generative Networks (2020, background): “A concurrent work [2] proposes that surrogate network with spatial transformers could be used to partially alleviates this aforementioned distributional shift at test-time.”
Anomalous Instance Detection in Deep Learning: A Survey (2020, background): “On the other hand, [67] use a generative model trained on benign images to estimate the manifold.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “…including ﬁne-tuning network weights for each image [7, 38, 46], choosing better or multiple layers to project and edit [1, 2, 19, 60], designing better encoders [45, 57], modeling image corruption and transformations [5, 24], and discovering meaningful latent directions [49, 16, 27, 21].”
Deep Learning Techniques for Inverse Problems in Imaging (2020, background): “the range of a generator, and was also independently proposed in [139]–[141].”