Adversarial Feature Matching for Text Generation

By Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, Lawrence Carin. 2017

363 citations recorded by the source corpus.

The Generative Adversarial Network (GAN) has achieved great success in generating realistic (real-valued) synthetic data. However, convergence issues and difficulties dealing with discrete data hinder the applicability of GAN to text. We propose a framework for generating realistic text via adversarial training. We employ a long short-term memory network as generator, and a convolutional network as discriminator. Instead of using the standard objective of GAN, we propose matching the high-dimensional latent feature distributions of real and synthetic sentences, via a kernelized discrepancy metric. This eases adversarial training by alleviating the mode-collapsing problem. Our experiments show superior performance in quantitative evaluation, and demonstrate that our model can generate realistic-looking sentences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “However, the reward in this case can be extremely sparse (as discussed in Section 3.2), yielding to high-variance gradient estimation, which is known to be challenging for optimization [57].”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “, 2014) to incorporate feedback signals for text generation (Yu et al., 2017; Lin et al., 2017; Zhang et al., 2017; Guo et al., 2017; Fedus et al., 2018; Zhang et al., 2018).”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “For the generator, instead of performing multinomial sampling at each generating step and leveraging REINFORCE-like method as in [7], we clamp all the randomness in the generation process to an initial input noise vector, and employ a discrete approximation strategy as used in [12].”
Texygen: A Benchmarking Platform for Text Generation Models (2018, methodology): “In the current version of Texygen, we implement various likelihood-based models such as vanilla MLE language model, SeqGAN [16], MaliGAN [3], RankGAN [10], TextGAN (feature matching) [17], GSGAN (GAN with Gumbel Softmax trick) [8] and LeakGAN [5].”
Adversarial Text Generation Without Reinforcement Learning (2018, methodology): “Zhang et al. (2017b) use a soft-argmax approximation with a convolutional network discriminator to smooth policy gradients to the generator, and pre-train the discriminator on sentence permutations to speed-up convergence.”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, background): “…et al., 2015), to more sophisticated compositional functions such as Recurrent Neural Networks (RNNs) (Tai et al., 2015; Sutskever et al., 2014), Convolutional Neural Networks (CNNs) (Kalchbrenner et al., 2014; Kim, 2014; Zhang et al., 2017a) and Recursive Neural Networks (Socher et al., 2011a).”
Long Text Generation via Adversarial Training with Leaked Information (2017, methodology): “Inspired by ranking idea from RankGAN (Lin et al. 2017), we propose a simple, time-efﬁcient, rank-based method to rescale the re-wards, named as bootstrapped rescaled activation .”
Language GANs Falling Short (2018, background): “Notably, RankGAN (Lin et al., 2017), MaliGAN (Che et al., 2017), TextGAN (Zhang et al., 2017), LeakGAN (Guo et al., 2017) and IRL-GAN (Shi et al., 2018) were proposed soon after.”
Deconvolutional Latent-Variable Model for Text Sequence Matching (2017, methodology): “To investigate the effectiveness of our latent-variable model, we first train it in an unsupervised manner, using the dataset in (Zhang et al. 2017a), where sentences from two corpora, i.e, BookCorpus dataset (Zhu et al. 2015) and the arXiv dataset, are merged together in equal proportion.”
Deconvolutional Paragraph Representation Learning (2017, background): “This deconvolutional module can also be leveraged as building block in VAE[30, 31] or GAN[32, 33]”
Adversarial Symmetric Variational Autoencoder (2017, background): “In the context of data synthesis, GANs have shown tremendous capabilities in generating realistic, sharp images from models that learn to mimic the structure of real data [3, 4, 5, 6, 7, 8].”
Triangle Generative Adversarial Networks (2017, methodology): “Various methods and model architectures have been proposed to improve and stabilize the training of GAN, such as feature matching [21, 22, 23], Wasserstein GAN [24] and energy-based GAN [25] among many other related works.”
Generative Adversarial Network Training is a Continual Learning Problem (2018, methodology): “We apply EWC and IS to textGAN (Zhang et al., 2017), a recently proposed model for text generation in which the discriminator uses feature matching to stabilize training.”
Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning (2019, methodology): “As a result, algorithms such as REINFORCE (Sutton et al. 2000) or those that approximate the discrete tokens with temperature-softmax probability vectors (Kusner and Hern´andez-Lobato 2016; Zhang et al. 2017) are used.”
TextBox: A Unified, Modularized, and Extensible Framework for Text Generation (2021, neutral): “, 2017) TextGAN (Zhang et al., 2017) RankGAN (Lin et al.”
LARA: Attribute-to-feature Adversarial Learning for New-item Recommendation (2020, background): “Moreover, it has already gained huge success in many tasks, such as image generation [22, 40], natural language generation [15, 34, 41], and music generation [9, 10].”
Multi-turn Dialogue Response Generation with Autoregressive Transformer Models (2019, background): “…Li et al. 2016a; Serban et al. 2016; Xing et al. 2017; Serban et al. 2017b; Serban et al. 2017a; Yu et al. 2017; Li et al. 2017; Che et al. 2017; Zhang et al. 2017; Xu et al. 2017; Zhang et al. 2018b; Li et al. 2016b; Naka-mura et al. 2019; Zhang et al. 2018a; Olabiyi et al. 2018; Olabiyi,…”