Learning from Simulated and Unsupervised Images through Adversarial Training

By Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, J. Susskind, Wenda Wang, Russ Webb. 2016

1,905 citations recorded by the source corpus.

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulators output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a self-regularization term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “Such biases can be somewhat alleviated through data augmentation, specifically, by harnessing images generated from 3D renderers [39, 4].”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Various methods have also been proposed to learn an image-to-image translation in the absence of training pairs [2, 33, 34, 46, 49, 51, 54, 63].”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “Pixel loss + GAN [42] Like our method, Shrivastava et al.”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “Some works enforce the translation to preserve certain properties of the source domain data, such as pixel values [21], pixel gradients [22], semantic features [10], class labels [22], or pairwise sample distances [16].”
A Hybrid Model for Identity Obfuscation by Face Replacement (2018, background): “In [25,26], rendered images are modified to be more realistic by means of adversarial training.”
Visual Feature Attribution Using Wasserstein GANs (2017, methodology): “Wasser-stein GANs are a modiﬁcation to the classic GAN paradigm where the discriminator is replaced by a critic which does not have an activation function in its ﬁnal layer and which is constrained to be a K -Lipschitz function.”
Faster AutoAugment: Learning Augmentation Strategies using Backpropagation (2019, background): “ operations (O 1;:::;O K), such as shear x and solarize. An operation O k operates a given image with probability p k and magnitude k. erate images that promote the performance of image classiﬁers. [27, 28] use GANs to modify the outputs of simulators to look like real objects. Automating data augm”
Fast AutoAugment (2019, background): “[32] employed a generative adversarial network (GAN) [9] to generate images that augment datasets.”
GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks (2018, background): “In [20], the authors use an adversarial network to improve the quality of simulated images, and use these for further training.”
Few-Shot Adversarial Domain Adaptation (2017, background): “Some try to find a function which can map from the source domain to the target domain [47, 28, 19, 16, 11, 55, 52].”
Catastrophic forgetting and mode collapse in GANs (2020, methodology): “[25] used a buffer of recent old fake samples to refine reasonably good fake samples.”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, background): “Since evaluating the quality of their predictive uncertainty is hard, deploying them in real-world systems raises serious concerns in AI Safety (Amodei et al., 2016), e.g., one can easily break a secure authentication system that can be unlocked by detecting the gaze and iris of eyes using DNNs (Shr”
Adversarial Audio Synthesis (2018, background): “Firstly, GANs could be useful for data augmentation (Shrivastava et al., 2017) in data-hungry speech recognition systems.”