Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking

By Tsung-Hsien Wen, Milica Gasic, Dongho Kim, Nikola Mrksic, Pei-hao Su, David Vandyke, Susannah Young. 2015

187 citations recorded by the source corpus.

The natural language generation (NLG) component of a spoken dialogue system (SDS) usually needs a substantial amount of handcrafting or a well-labeled dataset to be trained on. These limitations add significantly to development costs and make cross-domain, multi-lingual dialogue systems intractable. Moreover, human languages are context-aware. The most natural response should be directly learned from data rather than depending on predefined syntaxes or rules. This paper presents a statistical language generator based on a joint recurrent and convolutional neural network structure which can be trained on dialogue act-utterance pairs without any semantic alignments or predefined grammar trees. Objective metrics suggest that this new model outperforms previous methods under the same experimental conditions. Results of an evaluation by human judges indicate that it produces not only high quality but linguistically varied utterances which are preferred compared to n-gram and rule-based systems.

Subfields: Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Generative Models

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, methodology): “A prime example is the delexicalization technique used by most current generators (e.g., Oh and Rudnicky, 2000; Mairesse et al., 2010; Wen et al., 2015a,b; Juraska et al., 2018): It is generally assumed that attribute (slot) values from the input meaning representation (MR) can be replaced by…”
Survey of Hallucination in Natural Language Generation (2022, methodology): “Following the previous works such as [36, 173, 187], Li et al.”
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems (2015, methodology): “As suggested in Wen et al. (2015), a backward reranker is introduced in Section 3.3 to improve ﬂuency.”
A Network-based End-to-End Trainable Task-oriented Dialogue System (2016, methodology): “This system action vector is then used to condition a response generation network (Wen et al., 2015a; Wen et al., 2015b) which generates the required system output token by token in skeletal form.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “…language generation (Langkilde and Knight, 1998; Oh and Rudnicky, 2000; Walker et al., 2002; Ratnaparkhi, 2002; Stent et al., 2004; Rieser and Lemon, 2010; Mairesse et al., 2010; Mairesse and Young, 2014; Wen et al., 2015; Sharma et al., 2016), and dialogue policy learning (Young et al., 2013).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “Recent research often adds a general, overall quality criterion (Wen et al. 2015b,a; Manishina et al. 2016; Novikova, Lemon, and Rieser 2016; Novikova et al. 2017), or even uses only that (Sharma et al. 2016a).”
Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings (2016, methodology): “We also showed that our generator can learn to produce meaningful utterances using a much smaller amount of training data than what is typically used for RNN-based approaches.”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, methodology): “A series of neural methods have been proposed to generate accurate, natural, and diverse utterances, including HLSTM (Wen et al., 2015a),”
Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models (2017, methodology): “Seq2seq models with attention (Bahdanau et al., 2014) parameterize the per-symbol conditional probability as: for 1 ≤ i ≤ N , where DecoderRNN() is a recurrent neural network that map the sequence of decoder symbols into ﬁxed-length vectors, and Attention() is a function that yields a ﬁxed-size vect”
Few-shot Natural Language Generation for Task-Oriented Dialog (2020, methodology): “With the blooming of deep learning, neural sequential models have shown powerful capability and ﬂexibility in NLG. Extensive efforts have been made, including new architecture choices such as RNNs (Wen et al., 2015a), attention RNNs (Duˇsek and Jurˇc´ıˇcek, 2016), SC-LSTM (Wen et al., 2015b) and its”
A Context-aware Natural Language Generator for Dialogue Systems (2016, methodology): “They are delexicalized for generation to avoid sparsity, i.e., stop names, vehicles, times, etc., are replaced by placeholders (Wen et al., 2015a).”
Attention with Intention for a Neural Network Conversation Model (2015, methodology): “Recent methods [15] relax such requirement to some extent but their whole systems ∗Presented at NIPS Workshop on Machine Learning for Spoken Language Understanding and Interaction 2015.”
RankME: Reliable Human Ratings for Natural Language Generation (2018, background): “This is one of the reasons why some more recent research adds a general, overall quality criterion (Wen et al., 2015a,b; Manishina et al., 2016; Novikova et al., 2016, 2017a), or even uses only that (Sharma et al., 2016).”
Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation (2017, methodology): “Research in the task oriented setting has mostly made use of automated metrics such as BLEU and human evaluation (Wen et al., 2015b; Sharma et al., 2017; Dušek and Jurcicek, 2016).”
Natural Language Generation in Dialogue using Lexicalized and Delexicalized Data (2016, methodology): “We give a summary of the baseline models we use for performance comparisons in Section 4.7.”
Findings of the E2E NLG Challenge (2018, methodology): “…costly semantic alignment between meaning representations (MRs) and the corresponding natural language reference texts, e.g. (Duˇsek and Jurˇc´ıˇcek, 2015; Wen et al., 2015b; Mei et al., 2016; Wen et al., 2016; Sharma et al., 2016; Duˇsek and Jurˇc´ıˇcek, 2016a; Lampouras and Vlachos, 2016).”
Neural Belief Tracker: Data-Driven Dialogue State Tracking (2016, background): “The following example shows the true state after each user utterance in a three-turn conversation: User: I’m looking for a cheap restaurant inform(price=cheap) System: How about Thai food?”
Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems (2016, background): “In the above scenarios, the dialogs are domain-speciﬁc, and a typical approach to such in-domain systems is by human engineering, for example, using manually constructed ontologies [23], natural language templates [20], and even predeﬁned dialog states [25].”