Adversarial Generation of Natural Language

By Sandeep Subramanian, Sai Rajeswar, Francis Dutil, C. Pal, Aaron C. Courville. 2017

210 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have gathered a lot of attention from the computer vision community, yielding impressive results for image generation. Advances in the adversarial generation of natural language from noise however are not commensurate with the progress made in generating images, and still lag far behind likelihood based methods. In this paper, we take a step towards generating natural language with a GAN objective alone. We introduce a simple baseline that addresses the discrete output space problem without relying on gradient estimators and show that it is able to achieve state-of-the-art results on a Chinese poem generation dataset. We present quantitative results on generating sentences from context-free and probabilistic context-free grammars, and qualitative language modeling results. A conditional version is also described that can generate sequences conditioned on sentence characteristics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “Conditional text generation via GAN training has been explored in Rajeswar et al. (2017); Li et al. (2017).”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, background): “While in some cases, such as those presented in their paper, it might be plausible to generate counterfactual examples, in most cases in NLP it is still too hard to generate realistic texts with conditional generative models (Lin et al. 2017; Che et al. 2017; Subramanian et al. 2017; Guo et al. 2018”
Controllable Abstractive Summarization (2017, background): “This is exemplified by work focusing on natural language generation such as (Bowman et al., 2016; Yu et al., 2017; Zhao et al., 2017; Rajeswar et al., 2017).”
Long Text Generation via Adversarial Training with Leaked Information (2017, methodology): “To evaluate the performance of LeakGAN in short text generation, we pick the dataset of Chinese poems which is proposed by (Zhang and Lapata 2014) and most re-lated work such as (Yu et al. 2017; Rajeswar et al. 2017; Lin et al. 2017).”
Clinically Accurate Chest X-Ray Report Generation (2019, background): “Alternatively, Rajeswar et al. (2017) and Fedus et al. (2018) have tried using Generative Adversarial Neural Networks (GANs) for text generation.”
Optimizing distributions over molecular space. An Objective-Reinforced Generative Adversarial Network for Inverse-design Chemistry (ORGANIC) (2017, background): “There is also the possibility of augmenting this approach with the recent developments of attention networks and latent space exploration with text sequence models [45, 46].”
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2019, unknown): “…et al., 2015; Lamb et al., 2016; Ranzato et al., 2016; Yu et al., 2016; Zhu et al., 2018; Lu et al., 2018; Lin et al., 2017; Guo et al., 2017; Rajeswar et al., 2017; Wiseman and Rush, 2016; Nie et al., 2019; Shi et al., 2018; de Masson d’Autume et al., 2019; Rennie et al., 2016; Feng et al.,…”
Learning to Repair Software Vulnerabilities with Generative Adversarial Networks (2018, methodology): “For our second experiment, we generate data from a simple Context Free Grammar similar to that used by Rajeswar et al. [24].”
Sometimes We Want Ungrammatical Translations (2021, background): “Towards averting such scenarios, efforts along the lines of building robust models with adversarial training have been a common topic of study in natural language processing (Rajeswar et al., 2017; Wu et al., 2018).”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “For text, it is possible to create oracle training data from a fixed set of grammars and then evaluate generative models based on whether (or how well) the generated samples agree with the predefined grammar [124].”
Improved Natural Language Generation via Loss Truncation (2020, background): “GANs have been proposed to learn models that minimize distinguishability (Li et al., 2017; Rajeswar et al., 2017; Dai et al., 2017).”
Towards Text Generation with Adversarially Learned Neural Outlines (2018, background): “Approaches without maximum-likelihood pre-training that operate on continuous softmax outputs from the generator such as [19, 43, 42] have also shown promise, but apply mostly in artificial settings.”
Conditional WaveGAN (2018, unknown): “Usage of generative models in natural language processing is also in progress with notabld ﬁelds including text modelling [28, 21], dialogue generation [16], and neural machine translation [27].”
Evaluating Text GANs as Language Models (2018, methodology): “a generator using either a recurrent neural network (RNN) (Yu et al., 2017; Guo et al., 2018; Press et al., 2017; Subramanian et al., 2017), or a Convolutional Neural Network (CNN) (Gulrajani et al.”
TextKD-GAN: Text Generation Using Knowledge Distillation and Generative Adversarial Networks (2019, methodology): “MLE-based methods suffer from exposure bias [20], which means that at training time the model is exposed to gold data only, but at test time it observes its own predictions.”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “While the generation of images and text is highly explored [123], [124], [125], generating synthetic tabular data is a less frequent concern.”
Adversarial Learning of Semantic Relevance in Text to Image Synthesis (2018, methodology): “Subramanian et al. (2017) and Press et al. (2017) use curriculum learning for text generation from gradually increasing the length of character sequences in text as the training progresses.”
Adversarially Regularized Autoencoders (2017, background): “There have also been recent related approaches that work directly with the soft outputs from a generator (Gulrajani et al., 2017; Rajeswar et al., 2017; Shen et al., 2017; Press et al., 2017).”