A Hybrid Convolutional Variational Autoencoder for Text Generation

By Stanislau Semeniuta, Aliaksei Severyn, Erhardt Barth. 2017

267 citations recorded by the source corpus.

In this paper we explore the effect of architectural choices on learning a variational autoencoder (VAE) for text generation. In contrast to the previously introduced VAE model for text where both the encoder and decoder are RNNs, we propose a novel hybrid architecture that blends fully feed-forward convolutional and deconvolutional components with a recurrent language model. Our architecture exhibits several attractive properties such as faster run time and convergence, ability to better handle long sequences and, more importantly, it helps to avoid the issue of the VAE collapsing to a deterministic model.

Subfields: Artificial Intelligence, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Generative Models

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “We show that it allows optimizing the latent variables more effectively, resulting in a higher measured KL when compared to the RNN-VAE [2] and the Hybrid VAE [48].”
Generating functional protein variants with variational autoencoders (2020, methodology): “419 The decoder consisted of two components, similar to the decoder in a ‘hybrid’ autoregressive 420 VAE model developed for text [38]: an ‘upsampling’ component, which contained 3 layers of 421 transposed convolutions to ‘upsample’ the latent vector to a sequence of the same length as the 422 outpu”
Deconvolutional Paragraph Representation Learning (2017, methodology): “Recently, [42] proposed a hybrid model by coupling a convolutional-deconvolutional network with an RNN, where the RNN acts as decoder and the deconvolutional model as a bridge between the encoder (convolutional network) and decoder.”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “Thus, many proposed solutions to posterior collapse focus on weakening the generator by replacing it with a non-recurrent alternative (Yang et al., 2017; Semeniuta et al., 2017) or modifying the training objective (Zhao et al.”
On Accurate Evaluation of GANs for Language Generation (2018, background): “Firstly, as opposed to conventional Language Models and Variational Autoencoder (VAE) based models [7, 26] that can be relatively easily compared with one another through perplexity values, comparing GANs with these models is difﬁcult.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, background): “(Semeniuta et al., 2017) consider extensions of the recurrent VAE where the RNNs are replaced with feed-forward and convolutional networks.”
Adversarially Regularized Autoencoders (2017, methodology): “However there have been some recent successes through employing convolutional decoders (Yang et al., 2017; Semeniuta et al., 2017), training the latent representation as a topic model (Dieng et al., 2017; Wang et al., 2018), using the von Mises–Fisher distribution (Guu et al., 2017), and combining…”
Controllable Paraphrase Generation with a Syntactic Exemplar (2019, methodology): “In order to disentangle the syntactic and semantic aspects of paraphrase generation we learn an explicit latent variable model using a variational autoencoder (VAE) (Kingma and Welling, 2014), which is now commonly applied to text generation (Bowman et al., 2016; Miao et al., 2016; Semeniuta et al.,”
Generating Sentences from Disentangled Syntactic and Semantic Spaces (2019, background): “Variational auto-encoders (VAEs, Kingma and Welling, 2014) are widely used in language generation tasks (Serban et al., 2017; Kusner et al., 2017; Semeniuta et al., 2017; Li et al., 2018b).”
An Introduction to Variational Autoencoders (2019, unknown): “Another prominent application of VAEs is modeling of text and or sequential data (Bayer and Osendorfer, 2014; Bowman et al., 2015; Serban et al., 2017; Johnson et al., 2016; Karl et al., 2017; Fraccaro et al., 2016; Miao et al., 2016; Semeniuta et al., 2017; Zhao et al., 2017; Yang et al., 2017; Hu ”
Stock Movement Prediction from Tweets and Historical Prices (2018, methodology): “1976 where we adopt the KL term annealing trick (Bowman et al., 2016; Semeniuta et al., 2017) and add a linearly-increasing KL term weight λ ∈ (0, 1] to gradually release the KL regularization effect in the training procedure.”
Controllable Text Generation (2017, methodology): “We build our framework starting from variational auto-encoders (section 2) which have been used for text generation (Bowman et al., 2015; Semeniuta et al., 2017), where sentence ˆ x is generated conditioned on latent code z .”
Eval all, trust a few, do wrong to none: Comparing sentence generation models (2018, methodology): “To address some of the issues of training VAE models for text discussed by Bowman et al. (2016), Semeniuta et al. (2017) propose a hybrid architecture composed of a convolutional encoder and a decoder composed of a de-convolutional and an autoregressive layer (LSTM or ByteNet, Kalchbrenner et al.,…”
From Variational to Deterministic Autoencoders (2019, background): “It is therefore not surprising that learning generative models has gained momentum in applications like chemistry [16,25], NLP [8,46] and computer vision [9,48].”
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (2018, methodology): “Recently, algorithms have been introduced for image style transfer using generative models like Generative Ad-versarial Networks [4] (GAN) and Variational Auto En-coders [17] (VAE).”
Protein sequence design with deep generative models (2021, methodology): “Motivated by a similar model used for text generation [27], the decoder of the AR VAE contains an up-sampling component, which converts the compressed representation to the length of the output sequence, and an autoregressive component.”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, background): “VAE has successful applications in image generation [50], motion prediction [51], text generation [52], and expressive speech generation [53].”