A Neural Network Approach to Context-Sensitive Generation of Conversational Responses

By Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, William B. Dolan. 2015

912 citations recorded by the source corpus.

We present a novel response generation system that can be trained end to end on large quantities of unstructured Twitter conversations. A neural network architecture is used to address sparsity issues that arise when integrating contextual information into classic statistical models, allowing the system to take into account previous dialog utterances. Our dynamic-context generative models show consistent gains over both context-sensitive and non-context-sensitive Machine Translation and Information Retrieval baselines.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Scaling Neural Machine Translation (2018, methodology): “Similar sequence-to-sequence models are also applied to other natural language processing tasks, such as abstractive summarization (See et al., 2017; Paulus et al., 2018) and dialog (Sordoni et al., 2015; Serban et al., 2017; Dusek and Jurc´ıcek, 2016).”
LaMDA: Language Models for Dialog Applications (2022, background): “Dialog models [14, 15, 16], one of the most interesting applications of large language models, successfully take advantage of Transformers’ ability to represent long-term dependencies in text [17, 18].”
Towards a Human-like Open-Domain Chatbot (2020, background): “End-to-end neural network approaches (Shang et al., 2015; Vinyals and Le, 2015; Sordoni et al., 2015; Serban et al., 2016; Zhang et al., 2019), on the other hand, offer the simplicity of a single learned model.”
Wizard of Wikipedia: Knowledge-Powered Conversational agents (2018, background): “For example, popular chit-chat datasets such as Open-Subtitles (Vinyals & Le, 2015), Persona-Chat (Zhang et al., 2018) and Twitter (Sordoni et al., 2015) have tested the ability of sequence-to-sequence models that attend over the recent dialogue history, but do not attempt to recall long-term…”
Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “The advent of deep learning created interest in the construction of large-scale dialog datasets (Lowe et al., 2015b; Ritter et al., 2010; Sordoni et al., 2015) leading to the development of several end-to-end conversation systems (Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016; Serban et…”
A Knowledge-Grounded Neural Conversation Model (2017, methodology): “Based on (Sukhbaatar et al., 2015) we have: (2) Where A, C ∈ R d × v are the parameters of the memory network.”
A Diversity-Promoting Objective Function for Neural Conversation Models (2015, methodology): “…together with the * The entirety of this work was conducted at Microsoft. capacity to implicitly learn semantic and syntactic relations between pairs, and to capture contextual dependencies (Sordoni et al., 2015) in a way not possible with conventional SMT approaches (Ritter et al., 2011).”
Learning Through Dialogue Interactions (2016, background): “These include the chit-chat type end-to-end dialogue systems (Vinyals & Le, 2015; Li et al., 2015; Sordoni et al., 2015), which directly generate a response given the previous history of user utterance.”
Dialog-based Language Learning (2016, background): “In machine learning, one line of research has focused on supervised learning from dialogs using neural models [22, 4].”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, background): “2018), open-domain dialogue response generation (Sordoni et al. 2015) and abstractive summarization (See et al.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “Sequence-to-sequence (seq2seq) models (Sutskever et al., 2014) and their extensions (Lu-ong et al., 2015; Sordoni et al., 2015; Li et al., 2015), which are used for neural machine translation (MT), have been widely adopted for dialog generation systems.”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “els based on Recurrent Neural Networks (RNNs) are widely applied to response generation for dialogue (Vinyals and Le, 2015; Shang et al., 2015; Sordoni et al., 2015; Li et al., 2015).”
A Neural Conversational Model (2015, background): “More recently, work by Sordoni et al. (Sordoni et al., 2015) and Shang et al. (Shang et al., 2015), used recurrent neural networks to model dialogue in short conversations (trained on Twitter-style chats).”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, methodology): “There has been some work on using a larger set of automatically retrieved plausible responses when evaluating with BLEU (Sordoni et al., 2015).”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, methodology): “Amongst the simplest of statistical systems that can be used in this domain, that are based on data rather than hand-coding, are information retrieval models (Sordoni et al., 2015), which retrieve and rank responses based on their matching score with the recent dialogue history.”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “…variants of our HRED model, and compare against several alternatives, including basic n -gram models (Goodman 2001), a standard (non-hierarchical) RNN trained on the concatenation of the utterances in each triple, and a context-sensitive model (DCGM-I) recently proposed by Sordoni et al. (2015b).”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “Sordoni et al. (2015) improved Ritter et al.’s system by rescoring the outputs of a phrasal MT-based conversation system with a neural model incorporating prior context.”