Evaluating Text GANs as Language Models

By Guy Tevet, Gavriel Habib, Vered Shwartz, Jonathan Berant. 2018

33 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of “exposure bias”. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “Recent work has shown that when both quality and diversity is considered, GAN-generated text is substantially worse than language model generations (Caccia et al., 2018; Tevet et al., 2018; Semeniuta et al., 2018).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Consistently with previous works [39, 43, 23], we observe that the model, under the default exploration (i.e. GAN T =1 ), performs strictly worse than MLE.”
Text Generation by Learning from Demonstrations (2020, unknown): “, 2017) and the overall generation results are worse than MLE-trained models (Tevet et al., 2019; Caccia et al., 2020).”
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2019, methodology): “Tevet et al. (2018) proposes a method for approximating a distribution over tokens from GANs, and then evaluates models with standard LM metrics.”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “See here, here, Iqbal and Qureshi (2020), and Tevet et al. (2018) for some other measures for evaluating text generation models.”
On the Weaknesses of Reinforcement Learning for Neural Machine Translation (2019, background): “Another popular use of RL is for training GANs (Yang et al., 2018; Tevet et al., 2018).”
TextGAIL: Generative Adversarial Imitation Learning for Text Generation (2020, background): “Nevertheless, there are increasing criticisms (Caccia et al. 2020; Tevet et al. 2019; Semeniuta, Severyn, and Gelly 2018) of text GANs showing that GAN generated text is substantially worse than the text generated by MLE.”
Training language GANs from Scratch (2019, background): “Even with adversarial fine-tuning playing a limited role, extensive evaluation has shown that existing language GANs do not improve over maximum likelihood-trained models [19, 20, 21].”