DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder

By Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, Sunghun Kim. 2018

134 citations recorded by the source corpus.

Variational autoencoders~(VAEs) have shown a promise in data-driven conversation modeling. However, most VAE conversation models match the approximate posterior distribution over the latent variables to a simple prior such as standard normal distribution, thereby restricting the generated responses to a relatively simple (e.g., unimodal) scope. In this paper, we propose DialogWAE, a conditional Wasserstein autoencoder~(WAE) specially designed for dialogue modeling. Unlike VAEs that impose a simple distribution over the latent variables, DialogWAE models the distribution of data by training a GAN within the latent variable space. Specifically, our model samples from the prior and posterior distributions over the latent variables by transforming context-dependent random noise using neural networks and minimizes the Wasserstein distance between the two distributions. We further develop a Gaussian mixture prior network to enrich the latent space. Experiments on two popular datasets show that DialogWAE outperforms the state-of-the-art approaches in generating more coherent, informative and diverse responses.

Subfields: Natural Language Processing, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Generative Models

In Multimodal Learning

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, background): “Recent works also explore new frameworks such as adversarial learning (Li et al., 2017; Zhang et al., 2018c), variational autoencod-ing (Zhao et al., 2017; Gu et al., 2019), or both (Gao et al., 2019b) at the cost of added complexity and less scalability.”
Towards Coherent and Cohesive Long-form Text Generation (2018, neutral): “Nevertheless, we report the scores of widely used metrics, including negative log-likelihood (NLL), perplexity (PPL), BLEU and the proportion of unique n -grams within a single generation (intra-unique-n ), and across generations (inter-unique-n ), as in Gu et al. (2018).”
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling (2018, methodology): “Greedy search Greedy search has been the search algorithm of choice among the recent papers on neural dialogue modeling (Gu et al., 2018; Zhao et al., 2017; Xu et al., 2017; Weston et al., 2018; Zhang et al., 2018).”
Secure Deep Graph Generation with Link Differential Privacy (2020, methodology): “Moreover, to improve the capturing of global graph structures, we replace the direct binary crossentropy (BCE) loss on graph adjacency matrices in GVAE with a structure-oriented graph discriminator based on GCN and the framework of VAEGAN [Gu et al., 2019].”
Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization (2021, methodology): “Therefore, the higher Diversity scores, i.e., Dist-1/2/3 and Intra-1/2/3, further reflect the effectiveness of our proposed method.”
Importance of a Search Strategy in Neural Dialogue Modelling (2018, methodology): “…one of the most widely studied approaches to dialogue modelling (see, e.g., Serban et al., 2016, 2017; Zhao et al., 2017; Xu et al., 2017; Li et al., 2016, 2017a,b; Zemlyanskiy and Sha, 2018; Zhang et al., 2018; Miller 1https://beamdream.github.io/ et al., 2017; Shen et al., 2018; Gu et al., 2018).”
Mixture Content Selection for Diverse Sequence Generation (2019, background): “…such as language modeling (Bowman et al., 2016), machine translation (Zhang et al., 2016; Su et al., 2018; Deng et al., 2018; Shankar and Sarawagi, 2019), and conversation modeling (Serban et al., 2017; Wen et al., 2017; Zhao et al., 2017; Park et al., 2018; Wen and Luong, 2018; Gu et al., 2019).”
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable (2019, methodology): “To alleviate the issue of posterior collapse in VAE, some extension approaches are further developed, including conditional Wasserstein auto-encoder of DialogWAE (Gu et al., 2019) and implicit feature learning of iVAEMI (Fang et al.”
Conditional Structure Generation through Graph Variational Generative Adversarial Nets (2019, methodology): “Since CONDGEN and GVAE are basically a simple GCN model encapsulated in a VAEGAN and VAE framework, respectively, we also find that the memory consumptions of CONDGEN and GVAE are orders of magnitudes lower than GraphRNN and NetGAN.”
Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight (2020, methodology): “Following Gu et al. (2019), we also report Intra-{1,2,3} metrics which are computed as the average of distinct values within each sampled response.”
Synthetic Data for Deep Learning (2019, background): “[709] and has been extended into text style transfer by disentangling style and content in the latent space [285], disentangling syntax and semantics [36], DialogWAE for dialog modeling [215], Bilingual-GAN able to generate parallel sentences in two languages [479], and other works.”
Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner (2018, methodology): “Similarly, (Zhao et al., 2017; Shen et al., 2018; Gu et al., 2018) use variants of variational autoencoder .”
Jointly Optimizing Diversity and Relevance in Neural Response Generation (2019, background): “However, as VAE/CVAE conversation models can be limited to a simple latent representations such as standard Gaussian distribution, Gu et al. (2018) proposed to enrich the latent space by leveraging a Gaussian mixture prior.”
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020, methodology): “…et al., 2016); ( ii ) SeqGAN: a GAN based model for sequence generation (Li et al., 2017b); ( iii ) CVAE baseline (Zhao et al., 2017); ( iv ) Dialogue WAE, a conditional Wasserstein auto-encoder for response generation (Gu et al., 2019); ( v ) : A hierarchical VAE model (Serban et al., 2017).”