Language Generation with Recurrent Generative Adversarial Networks without Pre-training

By Ofir Press, Amir Bar, Ben Bogin, Jonathan Berant, Lior Wolf. 2017

105 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have shown great promise recently in image generation. Training GANs for language generation has proven to be more difficult, because of the non-differentiable nature of generating text with recurrent neural networks. Consequently, past work has either resorted to pre-training with maximum-likelihood or used convolutional networks for generation. In this work, we show that recurrent neural networks can be trained to generate text with GANs from scratch using curriculum learning, by slowly teaching the model to generate sequences of increasing and variable length. We empirically show that our approach vastly improves the quality of generated sequences compared to a convolutional baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Natural Language Generation

Citations in the corpus

On the Weaknesses of Reinforcement Learning for Neural Machine Translation (2019, background): “As work on language generation empirically showed, RNNs quickly learn to output very peaky distributions (Press et al., 2017).”
Adversarially Regularized Autoencoders (2017, background): “There have also been recent related approaches that work directly with the soft outputs from a generator (Gulrajani et al., 2017; Rajeswar et al., 2017; Shen et al., 2017; Press et al., 2017).”
TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks (2019, unknown): “In [19], RNN is trained to generate text with GAN using curriculum learning.”
Deep Learning for Sensor-based Human Activity Recognition (2020, background): “Variants of GANs has been applied to different fields such as language generation [113] and image generation [193].”
Towards Text Generation with Adversarially Learned Neural Outlines (2018, background): “Approaches without maximum-likelihood pre-training that operate on continuous softmax outputs from the generator such as [19, 43, 42] have also shown promise, but apply mostly in artiﬁcial settings.”
Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning (2020, background): “In the case of the harder-to-recognise minority class this leads to synthetic samples that are much easier to recognise than the real minority samples, but the distributions for the majority class show that the generator makes the samples just recognisable enough to avoid the AC loss being applied.”
Improved Training with Curriculum GANs (2018, methodology): “…synthetic data across a wide diversity of data modalities, including natural images (Radford et al., 2015; Karras et al., 2017), natural language (Yu et al., 2016; Press et al., 2017; Fedus et al., 2018), medical records (Esteban et al., 2017) and molecules (Sanchez-Lengeling et al., 2017).”
Evaluating Text GANs as Language Models (2018, methodology): “be gained either by using a noise vector as the initial state h0 (Press et al., 2017), or by sampling from the GAN’s internal distribution over possible output tokens (Yu et al.”
Learning to Repair Software Vulnerabilities with Generative Adversarial Networks (2018, methodology): “The first approach is to (linearly and deterministically) embed each one-hot vector into a lower dimensional space [23].”
Adversarial Learning of Semantic Relevance in Text to Image Synthesis (2018, background): “Subramanian et al. (2017) and Press et al. (2017) use curriculum learning for text generation from gradually increasing the length of character sequences in text as the training progresses.”
Training language GANs from Scratch (2019, background): “[55], ScratchGAN is a word level model and does not require teacher helping, curriculum learning or continuous relaxations during training.”