Detecting Overfitting of Deep Generative Networks via Latent Recovery

By Ryan Webster, Julien Rabin, Loïc Simon, Frédéric Jurie. 2019

107 citations recorded by the source corpus.

State of the art deep generative networks have achieved such realism that they can be suspected of memorizing training images. It is why it is not uncommon to include visualizations of training set nearest neighbors, to suggest generated images are not simply memorized. We argue this is not sufficient and motivates studying overfitting of deep generators with more scrutiny. We address this question by i) showing how simple losses are highly effective at reconstructing images for deep generators ii) analyzing the statistics of reconstruction errors for training versus validation images. Using this methodology, we show that pure GAN models appear to generalize well, in contrast with those using hybrid adversarial losses, which are amongst the most widely applied generative methods. We also show that standard GAN evaluation metrics fail to capture memorization for some deep generators. Finally, we note the ramifications of memorization on data privacy. Considering the already widespread application of generative networks, we provide a step in the right direction towards the important yet incomplete picture of generative overfitting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Deduplicating Training Data Makes Language Models Better (2021, methodology): “In computer vision, memorization of training data has been studied from various angles for both discriminative and generative models (e.g. Arpit et al., 2017; Webster et al., 2019; Feldman and Zhang, 2020; Stephenson et al., 2021; Teter-wak et al., 2021).”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, methodology): “Such embedding algorithms can be used to analyze the limitations of GANs [5], do image inpainting [8, 39, 38, 36], local image editing [40, 17], global image transformations such as image morphing and expression transfer [1], and few-shot video generation [35, 34].”
Source Generator Attribution via Inversion (2019, methodology): “We also tested the LBFGS optimization algorithm, which uses 2nd-order derivatives to accelerate convergence; similar to [23], we found that signiﬁcantly fewer optimization steps were required to invert an image, but we saw no signiﬁcant beneﬁt in the minimum loss obtained by the optimizer.”
Regularizing Generative Adversarial Networks under Limited Data (2021, background): “The success of the GAN methods heavily relies on a large amount of diverse training data which is often labor-expensive or cumbersome to collect [76].”
Revisiting Precision and Recall Definition for Generative Model Evaluation (2019, unknown): “In (Metz et al., 2017; Webster et al., 2019) test set samples ( i.e. those unseen during training), are recovered via optimization.”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, background): “Examples include various image and video editing tasks, image inpainting [13, 47, 46, 41], local image editing [42, 2], low bit-rate video conferencing [40], image super resolution [28, 12], image colorization [4, 27], and extracting 3D models [30].”
On Memorization in Probabilistic Deep Generative Models (2021, methodology): “A related approach to recovering training images is described in [32], using an optimization algorithm that identifies for every observation the closest sample that can be generated by the network.”