On the Discrimination-Generalization Tradeoff in GANs

By Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, Xiaodong He. 2017

110 citations recorded by the source corpus.

Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Catastrophic forgetting and mode collapse in GANs (2020, background): “In this paper, we focus on the continual learning problem in the discriminator as many prior works have showed that the quality of a GAN mainly depends on its discriminator [16, 17, 18].”
On catastrophic forgetting and mode collapse in Generative Adversarial Networks (2018, background): “In this paper, we focus on the continual learning problem in the discriminator as many prior works have showed that the quality of a GAN mainly depends on its discriminator [16, 17, 18].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “[460] developed generalization bounds between the true distribution and learned distribution under different evaluation metrics.”
SIN: Semantic Inference Network for Few-Shot Streaming Label Learning (2022, background): “Proof: This can be achieved by the uniform concentration bounds developed in empirical process [69], [70] and statistical learning theory [71]–[73].”
Generalized Energy Based Models (2020, methodology): “In fact networks with a single neuron are enough, as shown in (Zhang et al., 2017, Theorem 2.3).”
Prescribed Generative Adversarial Networks (2019, background): “Fortunately, GAN s do not merely memorize the training data (Zhang et al., 2017; Arora et al., 2017).”
When can Wasserstein GANs minimize Wasserstein Distance? (2020, background): “The work [38] shows that one-layer neural networks with ReLU activation are dense in the space of Lipschitz functions, however, it does not provide any efficient sample complexity guarantee.”
Adversarial Learning with Local Coordinate Coding (2018, unknown): “Specifically, the former term RX (F) implies that the set of discriminator should be smaller to have better generalization ability, and also be large enough to be able to identify the data distribution, which is consistent with (Zhang et al., 2018).”