Variational Neural Machine Translation

By Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, Min Zhang. 2016

209 citations recorded by the source corpus.

Models of neural machine translation are often from a discriminative family of encoderdecoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational model to learn this conditional distribution for neural machine translation: a variational encoderdecoder model that can be trained end-to-end. Different from the vanilla encoder-decoder model that generates target translations from hidden representations of source sentences alone, the variational model introduces a continuous latent variable to explicitly model underlying semantics of source sentences and to guide the generation of target translations. In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound. Experiments on both Chinese-English and English- German translation tasks show that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machine translation baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, methodology): “The models include Seq2seq, Seq2tree, Graph2tree, and commonly-used non-pretrained (AttSeq2Seq (Bahdanau et al., 2014), LSTMVAE (Zhang et al., 2016), and Trans-former (Vaswani et al., 2017)) and pretrained models (GPT-2 (Radford et al., 2019), BERTGen (De-vlin et al., 2018), and RoBERTaGen (Liu et a”
Variational Lossy Autoencoder (2016, neutral): “A VAE is frequently interpreted as a regularized autoencoder (Kingma & Welling, 2013; Zhang et al., 2016), but the conditions under which it is guaranteed to autoencode (reconstruction being close to original datapoint) are not discussed.”
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks (2017, methodology): “Researchers have successfully applied GAN to generate promising images conditionally (Mirza & Osindero, 2014; Reed et al., 2016; Zhang et al., 2016b) and unconditionally (Radford et al., 2015; Nguyen et al., 2016), to realize image manipulation and super-resolution (Zhu et al., 2016; Sønderby et…”
Analyzing Uncertainty in Neural Machine Translation (2018, background): “Notable exceptions are latent variable models (Blunsom et al., 2008; Zhang et al., 2016) which explicitly attempt to model multiple modes in the data distribution, or decoding strategies which attempt to predict diverse outputs while leaving the model unchanged (Gimpel et al., 2013; Vijayakumar et…”
Improved Variational Autoencoders for Text Modeling using Dilated Convolutions (2017, background): “(Ser-ban et al., 2016; Zhang et al., 2016) applies variational inference to dialogue modeling and machine translation and found some improvement in terms of generated text quality, but no language modeling results are reported.”
Identifying and Controlling Important Neurons in Neural Machine Translation (2018, methodology): “ion in neural language representations. We would also like to develop more sophisticated ways to control translation output, for example by modifying representations in variational NMT architectures (Zhang et al., 2016; Su et al., 2018). ACKNOWLEDGMENTS This research was carried out in collaboration”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, methodology): “We demonstrate that mixture models, when successfully trained, consistently outperform variational NMT (Zhang et al., 2016) and diverse decoding algorithms such as diverse beam search (Vijayakumar et al., 2016; Li et al., 2017) and biased sampling (Graves, 2013; Fan et al., 2018).”
Deconvolutional Latent-Variable Model for Text Sequence Matching (2017, methodology): “The proposed framework is closely related to recent research on incorporating NVI into text modeling (Bowman et al. 2016b; Miao, Yu, and Blunsom 2016; Xu et al. 2017; Zhang et al. 2016; Serban et al. 2017).”
A Hybrid Convolutional Variational Autoencoder for Text Generation (2017, methodology): “Zhang et al. (2016) and Serban et al. (2016) apply VAE to sequence-to-sequence problems, improving over deterministic alternatives.”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, methodology): “This issue is mostly attributed to the decoding algorithm (Li et al., 2016), and recently to the model (Zhang et al., 2016; Schulz et al., 2018a).”
Dual Latent Variable Model for Low-Resource Natural Language Generation in Dialogue Systems (2018, methodology): “Inspired by work of Zhang et al. (2016), we also employ the Monte-Carlo method to approximate the expectation of the posterior in Eq.”
Sequence-to-Dependency Neural Machine Translation (2017, methodology): “These methods show promising improvement for SMT. Recently, neural machine translation (NMT) has achieved better performance than SMT in many language pairs (Luong et al., 2015a; Zhang et al., 2016; Shen et al., 2016; Wu et al., 2016; Neubig, 2016).”
Adversarial Domain Adaptation for Variational Neural Language Generation in Dialogue Systems (2018, background): “Zhang et al. (2016) introduced a variational neural machine translation that incorporated a continuous latent variable to model underlying semantics of sentence pairs.”
Latent Alignment and Variational Attention (2018, background): “Of particular interest are those that augment an RNN with latent variables (typically Gaussian) at each time step [17, 22, 66, 23, 40] and those that incorporate latent variables into sequence-to-sequence models [84, 7, 70, 64].”
Document-Level Neural Machine Translation with Hierarchical Attention Networks (2018, methodology): “e document-level SMT paradigm. Sentence-level NMT Initial studies on NMT enhanced the sentence-level context by using memory networks (Wang et al.,2016), self-attention (Miculicich Werlen et al.,2018;Zhang et al., 2016), and latent variables (Yang et al.,2017). Document-level NMT Tiedemann and Scher”
A Hierarchical Latent Structure for Variational Conversation Modeling (2018, background): “Recently, latent variable models such as VAEs have been adopted in language modeling (Bow-man et al., 2016; Zhang et al., 2016; Serban et al., 2017).”
Mixture Content Selection for Diverse Sequence Generation (2019, background): “…(Kingma and Welling, 2013) are used for diverse generation in several tasks, such as language modeling (Bowman et al., 2016), machine translation (Zhang et al., 2016; Su et al., 2018; Deng et al., 2018; Shankar and Sarawagi, 2019), and conversation modeling (Serban et al., 2017; Wen et al.,…”