Improved Variational Autoencoders for Text Modeling using Dilated Convolutions

By Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, Taylor Berg-Kirkpatrick. 2017

403 citations recorded by the source corpus.

Recent work on generative modeling of text has found that variational auto-encoders (VAE) incorporating LSTM decoders perform worse than simpler LSTM language models (Bowman et al., 2015). This negative result is so far poorly understood, but has been attributed to the propensity of LSTM decoders to ignore conditioning information from the encoder. In this paper, we experiment with a new type of decoder for VAE: a dilated CNN. By changing the decoder's dilation architecture, we control the effective context from previously generated words. In experiments, we find that there is a trade off between the contextual capacity of the decoder and the amount of encoding information used. We show that with the right decoder, VAE can outperform LSTM language models. We demonstrate perplexity gains on two datasets, representing the first positive experimental result on the use VAE for generative modeling of text. Further, we conduct an in-depth investigation of the use of VAE (with our new decoding architecture) for semi-supervised and unsupervised labeling tasks, demonstrating gains over several strong baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

Neural Discrete Representation Learning (2017, methodology): “This has been done for language modelling with LSTM decoders [4], and more recently with dilated convolutional decoders [42].”
Toward Controlled Generation of Text (2017, background): “Bowman et al. (2015); Tang et al. (2016); Yang et al. (2017) instead use VAEs without discriminators.”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “In one thread of research, [6], [14] weaken and restrict the capacity of a generative network to enable higher-quality learned representations.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “As for representation learning, there is a variety of applications, ranging over image interpolation [19], one-shot generalization [35], language models [45], speech transformation [3], and more.”
Accelerating Protein Design Using Autoregressive Generative Models (2019, neutral): “Though others have reported difﬁculty training these models (Killoran et al., 2017), new machine learning techniques may make these models practical in the future (Kim et al., 2018; Yang et al., 2017; van den Oord & Vinyals, 2017).”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, background): “For instance, Gururangan et al. (2019); Chen et al. (2018); Yang et al. (2017) leveraged variational auto encoders (VAEs) in a form of sequence-to-sequence modeling on text classiﬁcation and sequential labeling.”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, background): “Despite some success (Bowman et al., 2016; Yang et al., 2017), predicting continuous latent representations does not work as well as the discrete case in our setting.”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, methodology): “We follow (Shen et al., 2020) and evaluate text in-ﬁlling performance on the Yahoo Answers dataset (Yang et al., 2017), which contains 100K/10K/10K documents for train/valid/test respectively.”
Discrete Autoencoders for Sequence Models (2018, methodology): “In this work, the study of text autoencoders (Bowman et al., 2016; Yang et al., 2017) is combined with the research on discrete autoencoders (Jang et al., 2016; Maddison et al., 2016).”
Deconvolutional Latent-Variable Model for Text Sequence Matching (2017, methodology): “Challenges with the NVI framework for text Extracting sentence features for text with the above NVI framework has been shown to be difficult (Bowman et al. 2016b; Yang et al. 2017).”
InfoVAE: Information Maximizing Variational Autoencoders (2017, background): “ on multiple performance metrics. 1. Introduction Generative models have shown great promise in modeling complex distributions such as natural images and text (Radford et al., 2015; Zhu et al., 2017; Yang et al., 2017; Li et al., 2017). These are directed graphical models which represent the joint d”
Deconvolutional Paragraph Representation Learning (2017, methodology): “In practice, we adopt a scheduled annealing strategy for α as in [36, 37], rather than fixing it a priori as in [1].”
Blank Language Models (2020, methodology): “Dataset We experiment on the Yahoo Answers dataset (Yang et al., 2017), which has 100k training documents and 10k documents for validation and testing respectively.”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “6 We evaluate our method on density estimation for text on the Yahoo and Yelp corpora (Yang et al., 2017) and images on OMNIGLOT (Lake et al.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, methodology): “This approach was also extended to utilize a convolutional encoder and decoder with dilated convolutions in (Yang et al., 2017).”
TextBox: A Unified, Modularized, and Extensible Framework for Text Generation (2021, neutral): “, 2016) CNN-VAE (Yang et al., 2017) Hybrid-VAE (Semeniuta et al.”
Learning to Represent Edits (2018, background): “Most (variational) autoencoders focus on encoding images but there have been works that autoencode sequences, such as text (Dai & Le, 2015; Yang et al., 2017; Bowman et al., 2015) and graphs (Simonovsky & Komodakis, 2018; Liu et al.”