Stochastic Wasserstein Autoencoder for Probabilistic Sentence Generation
By Hareesh Bahuleyan, Lili Mou, Hao Zhou, Olga Vechtomova. 2018
23 citations recorded by the source corpus.
The variational autoencoder (VAE) imposes a probabilistic distribution (typically Gaussian) on the latent space and penalizes the Kullback-Leibler (KL) divergence between the posterior and prior. In NLP, VAEs are extremely difficult to train due to the problem of KL collapsing to zero. One has to implement various heuristics such as KL weight annealing and word dropout in a carefully engineered manner to successfully train a VAE for text. In this paper, we propose to use the Wasserstein autoencoder (WAE) for probabilistic sentence generation, where the encoder could be either stochastic or deterministic. We show theoretically and empirically that, in the original WAE, the stochastically encoded Gaussian distribution tends to become a Dirac-delta function, and we propose a variant of WAE that encourages the stochasticity of the encoder. Experimental results show that the latent space learned by WAE exhibits properties of continuity and smoothness as in VAEs, while simultaneously achieving much higher BLEU scores for sentence reconstruction.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sebastian Gehrmann
- Hendrik Strobelt
- Inkit Padhi
- Aleksandra Mojsilovic
- Pin-Yu Chen
- Payel Das
- Vijil Chenthamarakshan
- Kahini Wadhawan
Citations in the corpus
- Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations (2021, methodology): “Many variations within the VAE family have therefore recently been proposed, such as WAE 73,74 and Adversarial Autoencoder 75 .”