Topic Augmented Neural Response Generation with a Joint Attention Mechanism

By Chen Xing, Wei Wu, Yuehua Wu, Jie Liu, Yalou Huang, M. Zhou, Wei-Ying Ma. 2016

49 citations recorded by the source corpus.

We consider incorporating topic information as prior knowledge into the sequence to sequence (Seq2Seq) network structure with attention mechanism for response generation in chatbots. To this end, we propose a topic augmented joint attention based Seq2Seq (TAJA-Seq2Seq) model. In TAJA-Seq2Seq, information from input posts and information from topics related to the posts are simultaneously embedded into vector spaces by a content encoder and a topic encoder respectively. The two kinds of information interact with each other and help calibrate weights of each other in the joint attention mechanism in TAJA2Seq2Seq, and jointly determine the generation of responses in decoding. The model simulates how people behave in conversation and can generate well-focused and informative responses with the help of topic information. Empirical study on large scale human judged generation results show that our model outperforms Seq2Seq with attention on both response quality and diversity.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, background): “…NLG solve in fact different tasks from ours: Chinese poetry generation (Zhang and Lapata, 2014; Yi et al., 2017; Wang et al., 2016), non-task-oriented response generation in chatbots (Xing et al., 2016, 2017), or morphological inﬂection (e.g. Faruqui et al., 2016; Kann and Sch¨utze, 2016).”
Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems (2016, background): “[12] and [27] introduce additional content (either the most mutually informative word or topic information) to the reply generator.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “A small improvement in decoder had achieved a better result in [94].”
Augmenting End-to-End Dialogue Systems With Commonsense Knowledge (2018, background): “(Xing et al. 2016) incorporated the topic words of the message obtained from a pre-trained latent Dirichlet allocation (LDA) model into the context vector through a joint attention mechanism.”
Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders (2017, background): “Other features should be extracted and provided to the models as conditionals in order to generate more specific responses (Xing et al., 2016; Li et al., 2016a); (2) the second category aims to improve the encoder-decoder model itself, including decoding with beam search and its variations (Wiseman ”
Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots (2016, unknown): “…2016), while the latter employs statistical machine translation techniques (Ritter et al., 2011) or the sequence to sequence framework (Shang et al., 2015; Serban et al., 2015; Vinyals and Le, 2015; Li et al., 2015; Li et al., 2016; Xing et al., 2016; Serban et al., 2016) to generate responses.”