An Improved Evaluation Framework for Generative Adversarial Networks

By Shaohui Liu, Yi Wei, Jiwen Lu, Jie Zhou. 2018

51 citations recorded by the source corpus.

In this paper, we propose an improved quantitative evaluation framework for Generative Adversarial Networks (GANs) on generating domain-specific images, where we improve conventional evaluation methods on two levels: the feature representation and the evaluation metric. Unlike most existing evaluation frameworks which transfer the representation of ImageNet inception model to map images onto the feature space, our framework uses a specialized encoder to acquire fine-grained domain-specific representation. Moreover, for datasets with multiple classes, we propose Class-Aware Frechet Distance (CAFD), which employs a Gaussian mixture model on the feature space to better fit the multi-manifold feature distribution. Experiments and analysis on both the feature level and the image level were conducted to demonstrate improvements of our proposed framework over the recently proposed state-of-the-art FID method. To our best knowledge, we are the first to provide counter examples where FID gives inconsistent results with human judgments. It is shown in the experiments that our framework is able to overcome the shortness of FID and improves robustness. Code will be made available.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Garrett Wilson
2. Ali Borji
3. Luc Van Gool
4. Diane Joyce Cook
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Pengchuan Zhang
8. Xiaogang Wang

In Generative Models

1. Diane Joyce Cook
2. Garrett Wilson
3. R. Devon Hjelm
4. Ali Borji
5. Shikhar Sharma
6. Yoshua Bengio
7. Tristan Sylvain
8. Pengchuan Zhang

In Evaluation & Benchmarks

1. Diane Joyce Cook
2. Garrett Wilson
3. Shikhar Sharma
4. Yoshua Bengio
5. Ali Borji
6. R. Devon Hjelm
7. Tristan Sylvain
8. Pengchuan Zhang

Citations in the corpus

• Object-Centric Image Generation from Layouts (2020, background): “While (Liu et al. 2018) introduce a class-aware version of FID, this is not applicable to our setting.”
• Pros and Cons of GAN Evaluation Measures (2018, background): “See also [61] for a class-aware version of FID.”
• A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “AdaBN[145] N not BN”
• Adversarial Transfer Learning (2018, background): “[73] highlight some cases where FID does not correlate well with human judgments but their proposed CAFD evaluation does.”
• Pros and Cons of GAN Evaluation Measures: New Developments (2021, methodology): “FID Liu et al. (2018) argue against the single-manifold Gaussian assumption in FID, and employ a Gaussian mixture model (GMM) to better fit the feature distribution and to include class information.”