On Accurate Evaluation of GANs for Language Generation

By Stanislau Semeniuta, Aliaksei Severyn, Sylvain Gelly. 2018

87 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “Recent work has shown that when both quality and diversity is considered, GAN-generated text is substantially worse than language model generations (Caccia et al., 2018; Tevet et al., 2018; Semeniuta et al., 2018).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Consistently with previous works [39, 43, 23], we observe that the model, under the default exploration (i.e. GAN T =1 ), performs strictly worse than MLE.”
Classification Accuracy Score for Conditional Generative Models (2019, methodology): “Procedurally similar to our method, [34] proposes a “reverse LM score”, which trains a language model on GAN data and tests on a real held-out set.”
Language GANs Falling Short (2018, methodology): “Concurrent with our work, Semeniuta et al. (2018) demonstrated the issues of local n-gram metrics.”
Evaluation of Text Generation: A Survey (2020, methodology): “, 2018) or detecting mode collapse (Semeniuta et al., 2019) in text generation with GAN (Goodfellow et al.”
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2019, methodology): “Semeniuta et al. (2018) proposes to use “Reverse Language Model score” or “Frechet InferSent Distance” to evaluate the model’s generation performance.”
Training language GANs from Scratch (2019, methodology): “Like ScratchGAN, SeqGAN-step [19] uses a recurrent discriminator to provide rewards per time step to a generator trained using policy gradient for unsupervised word level text generation.”
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, methodology): “In addition to the above point-wise metrics which score an individual sample, Semeniuta et al. (2019) proposed the Fréchet InferSent Distance (FID) to evaluate the model-level quality and diversity of generated samples, by computing the Fréchet distance between the Gaussian distribution ﬁtted to…”
A Discriminator Improves Unconditional Text Generation without Updating the Generator (2020, unknown): “It suffers from mode collapse (Semeniuta et al., 2018).”
Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study (2023, methodology): “Frechet Distance is a (3) Reduce the dimensionality of the embedding vectors using commonly used metric in Uniform Manifold Approximation and Projection (UMAP) benchmarking image generators [28] and has later been also applied to text embeddings [73].”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “However, self-BLEU has limitations in generating diverse output and detecting mode collapse in text generation with GANs [213].”
TextGAIL: Generative Adversarial Imitation Learning for Text Generation (2020, methodology): “ details of hyper-parameters are in the Appendix B. 5 Baselines For the baselines, we ﬁnd that the previous text GANs have much higher perplexity compared to GPT-2. Also, there are various criticisms [3, 26, 25] that point out text GANs are worse than MLEbased models. Therefore, for conditional gene”