Training Discriminative Models to Evaluate Generative Ones

By Timothée Lesort, Jean-François Goudou, David Filliat. 2018

19 citations recorded by the source corpus.

Generative models are known to be difficult to assess. Recent works, especially on generative adversarial networks (GANs), produce good visual samples of varied categories of images. However, the validation of their quality is still difficult to define and there is no existing agreement on the best evaluation process. This paper aims at making a step toward an objective evaluation process for generative models. It presents a new method to assess a trained generative model by evaluating the test accuracy of a classifier trained with generated data. The test set is composed of real images. Therefore, The classifier accuracy is used as a proxy to evaluate if the generative model fit the true data distribution. By comparing results with different generated datasets we are able to classify and compare generative models. The motivation of this approach is also to evaluate if generative models can help discriminative neural networks to learn, i.e., measure if training on generated data is able to make a model successful at testing on real settings. Our experiments compare different generators from the Variational Auto-Encoders (VAE) and Generative Adversarial Network (GAN) frameworks on MNIST and fashion MNIST datasets. Our results show that none of the generative models is able to replace completely true data to train a discriminative model. But they also show that the initial GAN and WGAN are the best choices to generate on MNIST database (Modified National Institute of Standards and Technology database) and fashion MNIST database.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning with Fully Probabilistic Models (2021, background): “Training and Evaluation Paradigms for CL In the context of CL, a wide range of training and evaluation paradigms are proposed, see [27, 28, 29, 30, 5, 31, 32].”
Generative Models from the perspective of Continual Learning (2018, methodology): “A different approach is to use labeled generated samples from a generator G (GAN or VAE) to train a classifier and evaluate it afterwards on real data [14].”
Continual Learning for Robotics (2019, methodology): “Another evaluation method is using generated data to train a classiﬁer and evaluate its accuracy on a test set of true data [85].”
Marginal Replay vs Conditional Replay for Continual Learning (2018, methodology): “We choose to evaluate two different models for the generator: WGAN-GP as used in [38] and the original GAN model [10] since it is a competitive baseline [23].”
An Investigation of Replay-based Approaches for Continual Learning (2021, methodology): “The RB measurement allows a better evaluation of all different sample selection strategies and is inspired by the Fitting Capacity measure for generator-based replay approaches [41].”