Attention with Intention for a Neural Network Conversation Model

By Kaisheng Yao, Geoffry Zweig, Baolin Peng. 2015

115 citations recorded by the source corpus.

In a conversation or a dialogue process, attention and intention play intrinsic roles. This paper proposes a neural network based approach that models the attention and intention processes. It essentially consists of three recurrent networks. The encoder network is a word-level model representing source side sentences. The intention network is a recurrent network that models the dynamics of the intention process. The decoder network is a recurrent network produces responses to the input from the source side. It is a language model that is dependent on the intention and has an attention mechanism to attend to particular source side words, when predicting a symbol in the response. The model is trained end-to-end without labeling data. Experiments show that this model generates natural responses to user inputs.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Diversity-Promoting Objective Function for Neural Conversation Models (2015, background): “Other researchers have subsequently sought to apply direct end-to-end Seq2Seq models (Shang et al., 2015; Vinyals and Le, 2015; Wen et al., 2015; Yao et al., 2015; Serban et al., 2016).”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “Neural response generation (Sordoni et al., 2015; Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016a; Wen et al., 2015; Yao et al., 2015; Luan et al., 2016; Xu et al., 2016; Wen et al., 2016; Li et al., 2016b; Su et al., 2016) is of growing interest.”
Neural Approaches to Conversational AI (2018, positive): “s true in dialogue data. Indeed, in dialogue entire spans of the source may not map to anything in the target and vice-versa.4 Some speciﬁc attention models for dialogue have been shown to be useful (Yao et al., 2015; Mei et al., 2017; Shao et al., 2017), e.g., to avoid word repetitions (which are d”
A Persona-Based Neural Conversation Model (2016, methodology): “Yao et al. (2015) employ an intention network to maintain the relevance of responses.”
Neural Text Generation from Structured Data with Application to the Biography Domain (2016, background): “, 2015), and modeling conversations and dialogues (Shang et al., 2015; Wen et al., 2015; Yao et al., 2015).”
Conversational Contextual Cues: The Case of Personalization and History for Response Ranking (2016, methodology): “Perplexity, BLEU, and deltaBLEU, adapted from language modeling and machine translation communities, have been used for evaluating novel responses (Yao et al., 2015; Sordoni et al., 2015; Gal-ley et al., 2015).”
Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning (2017, background): “…which map from an observable dialog history directly to a sequence of output words (Sordoni et al., 2015; Shang et al., 2015; Vinyals and Le, 2015; Yao et al., 2015; Serban et al., 2016; Li et al., 2016a,c; Luan et al., 2016; Xu et al., 2016; Li et al., 2016b; Mei et al., 2016; Lowe et al.,…”
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (2019, background): “The majority of work in dialogues is formulated as either open-domain dialogues (Shang et al., 2015; Vinyals and Le, 2015; Yao et al., 2015; Li et al., 2016a,b; Serban et al., 2017, 2016) or task-oriented dialogues (Henderson et al., 2014; Bor-des and Weston, 2016; Fatemi et al., 2016; Liu and…”
LSDSCC: a Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics (2018, methodology): “…methods employed by the existing NRG models can be categorized as: a) metrics inherited from Machine Translation, e.g., BLEU, Perplexity, etc. (Yao et al., 2015; Lowe et al., 2017; Wu et al., 2018); b) discrete scores measuring the quality of generated results by human labeling (Shang et al.,…”
Topic Augmented Neural Response Generation with a Joint Attention Mechanism (2016, background): “K. Yao (Yao et al., 2015) added an extra RNN between the encoder and the decoder with attention to conclude and explain intentions from posts.”
Generating Text from Structured Data with Application to the Biography Domain (2016, background): “…(Kiros et al., 2014; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Fang et al., 2015; Xu et al., 2015), machine translation (Devlin et al., 2014; Bahdanau et al., 2015; Luong et al., 2015), and modeling conversations and dialogues (Shang et al., 2015; Wen et al., 2015; Yao et al., 2015).”
Paying Attention to Attention: Highlighting Influential Samples in Sequential Analysis (2018, methodology): “(Yao et al., 2015) uses an encoder-decoder framework with attention to model conversations and generate natural responses to user input.”
A Theoretical Framework for Conversational Search (2017, background): “Conversational agents for more advanced multi-turn tasks have been proposed continuously since then, for instance recent work to isolate and resolve technical issues typically handled by a help desk [42].”