Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation

By Iulian Serban, Tim Klinger, G. Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, Aaron C. Courville. 2016

193 citations recorded by the source corpus.

We introduce a new class of models called multiresolution recurrent neural networks, which explicitly model natural language generation at multiple levels of abstraction. The models extend the sequence-to-sequence framework to generate two parallel stochastic processes: a sequence of high-level coarse tokens, and a sequence of natural language words (e.g. sentences). The coarse sequences follow a latent stochastic process with a factorial representation, which helps the models generalize to new examples. The coarse sequences can also incorporate task-specific knowledge, when available. In our experiments, the coarse sequences are extracted using automatic procedures, which are designed to capture compositional structure and semantics. These procedures enable training the multiresolution recurrent neural networks by maximizing the exact joint log-likelihood over both sequences. We apply the models to dialogue response generation in the technical support domain and compare them with several competing models. The multiresolution recurrent neural networks outperform competing models by a substantial margin, achieving state-of-the-art results according to both a human evaluation study and automatic evaluation metrics. Furthermore, experiments show the proposed models generate more fluent, relevant and goal-oriented responses.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Evaluation of Text Generation: A Survey (2020, methodology): “However, the most commonly used human evaluations of dialog systems is still via crowd-sourcing platforms such as Amazon Mechanical Turk (AMT) (Serban et al., 2016a; Peng et al., 2020; Li et al., 2020; Zhou et al., 2020).”
Assessing The Factual Accuracy of Generated Text (2019, background): “Recently, there has been wide empirical success in text summarization [15, 21, 27], machine translation [1, 36, 39], dialogue response generation [12, 28, 29], and other text generation tasks.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, methodology): “[57] extended this method by learning a sequence of the latent variables, where the latent variable of next utterance is only conditioned on the previous latent variable sequence.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “Ultimately the impact of any unnaturalness in the dialogues depends on the task and context in which the dialogue system is deployed.”
Multi-turn Dialogue Response Generation with Autoregressive Transformer Models (2019, methodology): “…limitations of the MLE objective alone (Li et al. 2016a; Li et al. 2016b; Li et al. 2017; Yu et al. 2017; Che et al. 2017; Zhang et al. 2017; Xu et al. 2017; Serban et al. 2017b; Zhang et al. 2018b; Olabiyi et al. 2018; Olabiyi, Khazan, and Mueller 2018; Olabiyi et al. 2019a; Olabiyi et al. 2019b).”
Neural Response Generation via GAN with an Approximate Embedding Layer (2017, unknown): “Earlier works focused on paired word sequences only, while Zhou et al. (2016) and Iulian et al. (2017) have demonstrated that the comprehensibility of the generated responses can benefit from multiview training with respect to words, coarse tokens and utterances.”
Generative Deep Neural Networks for Dialogue: A Short Review (2016, methodology): “The Latent Variable Hierarchical Recurrent Encoder-Decoder model (VHRED) [Serban et al., 2016c] is an HRED model with an additional component: a high-dimensional stochastic latent variable at every dialogue turn.”
If Beam Search Is the Answer, What Was the Question? (2020, background): “…and σ . λ and σ axes are log-scaled. or even approximate—the highest-scoring candidate under a model, it has repeatedly proven its merit in practice (Serban et al., 2017; Edunov et al., 2018; Yang et al., 2019) and, thus, has largely been tolerated—even embraced—as NLP’s go-to search heuristic.”
Survey on evaluation methods for dialogue systems (2019, methodology): “Current research relies on crowdsourcing for human evaluation [Serban et al., 2017c; Wen et al., 2017].”
Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings (2017, background): “Compared to Rule, the neural models tend to generate shorter utterances (Li et al., 2016b; Serban et al., 2017b).”
Modeling Multi-turn Conversation with Deep Utterance Aggregation (2018, background): “Multi-turn conversation modeling plays a key role in dialogue systems, either for generation-based (Serban et al., 2017b; Serban et al., 2017a; Zhou et al., 2017; Wu et al., 2018) or retrieval-based ones (Wu et al., 2017; Zhou et al., 2016) in which the latter is the focus of this paper.”
A Persona-Based Multi-turn Conversation Model in an Adversarial Learning Framework (2018, background): “For example, Serban et al. [3]– [5] and Xing et al. [6] introduce a Hierarchical Recurrent Encoder-Decoder (HRED) network architecture that combines a series of recurrent neural networks to capture long-term context state within a dialogue.”