Language GANs Falling Short

By Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, Laurent Charlin. 2018

240 citations recorded by the source corpus.

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

Defending Against Neural Fake News (2019, background): “Likewise, models that are trained conditioned on their own predictions might avoid exposure bias, however, these objectives often lead to low performance on language tasks (Caccia et al., 2018).”
The Curious Case of Neural Text Degeneration (2019, background): “Recent work has shown that when both quality and diversity is considered, GAN-generated text is substantially worse than language model generations (Caccia et al., 2018; Tevet et al., 2018; Semeniuta et al., 2018).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “To our knowledge, this is the ﬁrst time that MLE falls short [5, 23] w.r.t. GAN-based approaches for this task.”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “However, due to reward sparsity and mode collapse (Zhou et al., 2020), text GANs failed so far to be competitive with state-of-the-art models trained with teacher forcing on NLG tasks (Caccia et al., 2018; Clark et al., 2019), and are mostly evaluated on synthetic datasets.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Self-BLEU-n, a measure of text diversity on a distributional level across samples proposed by (Zhu et al., 2018), ensuring that policies don’t converge into limited number of sequences that satisfy the imposed constraints Caccia et al. (2020).”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “Caccia et al. (2018) recently examined the diversity and quality tradeoffs for different language model architectures on synthetic datasets.”
A Distributional Approach to Controlled Text Generation (2020, background): “However, the shortcomings in terms of sample diversity, of optimization techniques when training generative models for text, has recently been documented in (Caccia et al., 2020).”
Evaluation of Text Generation: A Survey (2020, methodology): “In Caccia et al. (2018), it was empirically demonstrated that, when used to sample from the model outputs that has perfect BLEU with the corpus, the generated sentences were grammatically correct, but lacked semantic and/or global coherence, concluding that the generated text has poor information…”
What Context Features Can Transformer Language Models Use? (2021, background): “Recent work has shown that other metrics, such as diversity of outputs, are important for evaluating the quality of LMs as models for language generation (Hashimoto et al., 2019; Caccia et al., 2020).”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, methodology): “We test four methods, including the deterministic beam search and three stochastic methods — temperature sampling (Ficler and Goldberg, 2017; Caccia et al., 2020), nucleus sampling (Holtzman et al., 2020), and top-k sampling (Fan et al., 2018).”
Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation (2020, background): “On the other hand, if we measure diversity based on responses randomly sampled from a model conditional on a single query, the response quality is generally low (Caccia et al., 2018).”
Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding (2023, unknown): “In our work, we use the stochastic beam search (Caccia et al., 2018; Kool et al., 2019; Meister et al., 2021), which combines beam search and temperature sampling.”
Step-unrolled Denoising Autoencoders for Text Generation (2021, result): “We also compare to several other baselines reported in (d’Autume et al., 2019; Caccia et al., 2020): Se-qGAN (Yu et al., 2017), LeakGAN (Guo et al., 2017), RankGAN (Lin et al., 2017), MaliGAN (Che et al., 2017) and autoregressive (AR) language models.”
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2019, background): “On the other hand, non-MLE training methods are still struggling to outperform the MLE baseline (Caccia et al., 2018; de Masson d’Autume et al., 2019).”
TextGAIL: Generative Adversarial Imitation Learning for Text Generation (2020, methodology): “Nevertheless, there are increasing criticisms (Caccia et al. 2020; Tevet et al. 2019; Semeniuta, Severyn, and Gelly 2018) of text GANs showing that GAN generated text is substantially worse than the text generated by MLE.”
Text Generation with Efficient (Soft) Q-Learning (2021, background): “Since text generation models intrinsically trade off diversity and quality [5, 18], we vary the generation diversity by generating samples via top-p sampling [19] with different p values, and plot the entailment rate and perplexity against diversity, respectively.”
On the Weaknesses of Reinforcement Learning for Neural Machine Translation (2019, background): “Pretraining and training were considered ﬁnished when BLEU did not increase in the development set for 10 consecutive evaluations, and evaluation was done every 1000 and 5000 for batches of size 100 and 256 for pretraining and training respectively.”
A Discriminator Improves Unconditional Text Generation without Updating the Generator (2020, unknown): “(Caccia et al., 2019) precisely use this paired metric by adjusting the softmax temperature, thereby drawing a curve rather than only plot a dot in the quality-diversity space.”
Do Massively Pretrained Language Models Make Better Storytellers? (2019, background): “However, evaluating one k gives an incomplete view of the generation system – several researchers have emphasized the importance of evaluating generation systems over the entire quality-diversity spectrum, rather than a single point on it (Caccia et al., 2018; Hashimoto et al., 2019).”