Generating Sentences from a Continuous Space

By Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, Samy Bengio. 2015

2,543 citations recorded by the source corpus.

The standard recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global sentence representation. In this work, we introduce and study an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences. This factorization allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features. Samples from the prior over these sentence representations remarkably produce diverse and well-formed sentences through simple deterministic decoding. By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model's latent sentence space, and present negative results on the use of the model in language modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

An efficient framework for learning sentence representations (2018, background): “Bowman et al. (2015) proposed a generative model of sentences based on a variational autoencoder.”
Residual Energy-Based Models for Text Generation (2020, background): “Several variants of auto-encoders have also been investigated for representing and generating text (Bowman et al., 2016; Zhao et al., 2018), but they have not shown signiﬁcant improvements in terms of perplexity and they have so far been applied to relatively small datasets only.”
A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “This decoder architecture is used in the editor and used in the VAE decoder without the attention mechanism.”
Generating Sentences by Editing Prototypes (2017, methodology): “Second, our parameterization avoids collapsing the latent code, which is a serious problem with variational autoencoders in practice (Bowman et al., 2016).”
Zero-Shot Text-to-Image Generation (2021, methodology): “Bowman et al. (2015) use a similar schedule based on the sigmoid function.”
Neural Discrete Representation Learning (2017, methodology): “This has been done for language modelling with LSTM decoders [4], and more recently with dilated convolutional decoders [42].”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “We compare our model against a LSTM language model [17], the RNN-VAE [2], and the Hybrid VAE [48], which couples a convolutional architecture with an LSTM decoder.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, positive): “Consistent with ﬁndings in Bowman et al. (2015) and Kaae Sønderby et al. (2016), we found that stochastic optimization with the unmodiﬁed lower bound objective often gets stuck in an undesirable stable equilibrium.”
Density estimation using Real NVP (2016, background): “Recent work in this line of research has taken advantage of recent advances in recurrent networks [51], in particular long-short term memory [26], and residual networks [25, 24] in order to learn state-of-the-art generative image models [61, 46] and language models [32].”
Score-based Generative Modeling in Latent Space (2021, neutral): “In principle LSGM could be used to model, for example, language [96, 97], music [98, 10], or molecules [99, 100].”
Symbolic Music Generation with Diffusion Models (2021, background): “Of particular interest to us are sequential autoencoders (Bowman et al., 2016; Roberts et al., 2019) which use long short-term memory cells (Hochreiter & Schmidhuber, 1997) to model temporal context in sequential data distributions.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
Adversarial Examples for Evaluating Reading Comprehension Systems (2017, neutral): “Bowman et al. (2016) and Li et al. (2017) used such a setup for sentence and dialogue generation, respectively.”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “This is a common problem of sequential VAEs that use loss terms to regularize the latent encoding [49].”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Unfortunately, VAEs with powerful decoders suffer from posterior collapse [6, 9, 18, 46]: they will often neglect to use the latent variables at all, because the regularisation term in the loss function explicitly discourages this.”
OpenNMT: Open-Source Toolkit for Neural Machine Translation (2017, background): “Recently work such as variational seq2seq auto-encoders (Bowman et al., 2016) or memory networks (We-”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “We trained a SEQ2SEQ model (Sutskever et al., 2014) with an attention mechanism (Bahdanau et al., 2015; Luong et al., 2015) on the OpenSubtitles dataset.”