SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

By Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu. 2016

2,590 citations recorded by the source corpus.

As a new way of training generative models, Generative Adversarial Net (GAN) that uses a discriminative model to guide the training of the generative model has enjoyed considerable success in generating real-valued data. However, it has limitations when the goal is for generating sequences of discrete tokens. A major reason lies in that the discrete outputs from the generative model make it difficult to pass the gradient update from the discriminative model to the generative model. Also, the discriminative model can only assess a complete sequence, while for a partially generated sequence, it is non-trivial to balance its current score and the future one once the entire sequence has been generated. In this paper, we propose a sequence generation framework, called SeqGAN, to solve the problems. Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the intermediate state-action steps using Monte Carlo search. Extensive experiments on synthetic data and real-world tasks demonstrate significant improvements over strong baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “SeqGAN (Yu et al., 2017) trains a language model by using policy gradients to train the generator to fool a CNN-based discriminator that discriminates between real and synthetic text.”
The Curious Case of Neural Text Degeneration (2019, background): “This is in part due to the recency bias and explanation-away problem, where language models have the tendency to rely overly on the short-term context that can easily explain away the longer-term context (Yu et al., 2017b).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Hence, Language GANs are based on gradient estimation via RL-based techniques [53].”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “the GAN training procedure can be unstable [9, 8, 63], lack a informative objective for measuring progress [4], and struggle with discrete data [102].”
Improved Training of Wasserstein GANs (2017, methodology): “Other attempts at language modeling with GANs [31, 14, 29, 5, 15, 10] typically use discrete models and gradient estimators [27, 12, 16].”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “Methods for controlling text generation can be categorized broadly into two types: training a model directly for controllable generation (Keskar et al., 2019; Ficler & Goldberg, 2017; Yu et al., 2017; Hu et al., 2017) or using a discriminator to guide generation (Holtzman et al.”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Controlled generation Current methods for controlled text generation involve either ﬁne-tuning existing models with Reinforcement Learning (RL) (Ziegler et al., 2019), training Generative Ad-versarial Networks (Yu et al., 2017), or training conditional generative models (Kikuchi et al., 2016; Ficler”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “, 2014) to incorporate feedback signals for text generation (Yu et al., 2017; Lin et al., 2017; Zhang et al., 2017; Guo et al., 2017; Fedus et al., 2018; Zhang et al., 2018).”
Neural Text Generation with Unlikelihood Training (2019, background): “The resulting distribution mismatch between training sequences and generated sequences is a known issue with maximum-likelihood training, motivating objectives that operate on model-generated sequences (Daum´e et al., 2009; Ross et al., 2011; Ranzato et al., 2015; Yu et al., 2016).”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Similar to approaches for generative adversarial NMT (Yu et al., 2017; Wu et al., 2017) we prefer a simpler convolutional architecture based on word embeddings for the human reward estimation.”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Several works (Ficler & Goldberg, 2017; Hu et al., 2017; Rajani et al., 2019; Yu et al., 2017; Ziegler et al., 2019; Keskar et al., 2019) have explored training or fine-tuning of models for controllable text generation.”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “Adversarial learning [6, 26] has been shown to be successful in dialog generation, translation, image captioning and a series of natural language generation tasks [7, 12, 27, 28, 29, 30, 31, 32, 33, 34].”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “A similar strategy is adopted in Yu et al. (2016a).”
Learning to Write with Cooperative Discriminators (2018, methodology): “(GANs), we use SeqGAN (Yu et al., 2017a), a GAN for discrete sequences trained with policy gradients.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, methodology): “Quality We follow Yu et al. (2017) by computing BLEU (Papineni et al., 2002) between the generations and the original data distributions to measure how similar the generations are.”
Controllable Abstractive Summarization (2017, background): “This is exempliﬁed by work focusing on natural language generation such as (Bowman et al., 2016; Yu et al., 2017; Zhao et al., 2017; Rajeswar et al., 2017).”
Texygen: A Benchmarking Platform for Text Generation Models (2018, methodology): “SeqGAN[16]adoptsadiscriminativemodelthatis trained to minimize the binary classification loss between real texts and generated texts.”