Topic Aware Neural Response Generation

By Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, M. Zhou, Wei-Ying Ma. 2016

490 citations recorded by the source corpus.

We consider incorporating topic information into a sequence-to-sequence framework to generate informative and interesting responses for chatbots. To this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The model utilizes topics to simulate prior human knowledge that guides them to form informative and interesting responses in conversation, and leverages topic information in generation by a joint attention mechanism and a biased generation probability. The joint attention mechanism summarizes the hidden vectors of an input message as context vectors by message attention and synthesizes topic vectors by topic attention from the topic words of the message obtained from a pre-trained LDA model, with these vectors jointly affecting the generation of words in decoding. To increase the possibility of topic words appearing in responses, the model modifies the generation probability of topic words by adding an extra probability item to bias the overall distribution. Empirical studies on both automatic evaluation metrics and human annotations show that TA-Seq2Seq can generate more informative and interesting responses, significantly outperforming state-of-the-art response generation models.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, methodology): “…approach to mitigating this problem is to use more sophisticated decoding algorithms, for instance with different forms of re-ranking (Li et al., 2016a; Shao et al., 2017) or conditioning on proﬁles, topics, and styles (Li et al., 2016b; Wang et al., 2017; Xing et al., 2017; Zhang et al., 2018b).”
Neural Generation for Czech: Data and Baselines (2019, background): “…NLG solve in fact different tasks from ours: Chinese poetry generation (Zhang and Lapata, 2014; Yi et al., 2017; Wang et al., 2016), non-task-oriented response generation in chatbots (Xing et al., 2016, 2017), or morphological inﬂection (e.g. Faruqui et al., 2016; Kann and Sch¨utze, 2016).”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “Xing et al. (2017) use topics as an additional input while decoding to produce more speciﬁc responses.”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “In particular, several works use control to tackle the same common sequence-to-sequence problems we address here (particularly genericness and unrelated output), in the context of single-turn response generation (Baheti et al., 2018; Li et al., 2016a, 2017a; Shen et al., 2017; Xing et al., 2017; Zha”
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot (2018, methodology): “The neural response generator in XiaoIce follows the sequence-to-sequence (seq2seq) framework [14, 15] used for conversation response generation [16, 17, 18, 19, 20, 21].”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, methodology): “Prior performance improvements to dialog models using topic information include appending topic as an additional input [23], or extracting topic information using Latent Dirichlet Allocation [24, 25].”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “r. Encoding richer information from query X ⊕C, such as longer dialog history [129], persona [62], hidden topics [121], has proved to be helpful for generating more informative responses. Xing et al. [160] extracted topic words, rather than hidden topic vectors, using LDA, and encoded such words in ”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, methodology): “the inherent attributes of dialogues explicitly is another way to improve the diversity of dialogues and ensures the consistency. Among diﬀerent attributes, topic and personality are widely explored. [95] noticed that people often associate their dialogues with topically related concepts and create ”
Learning to Control the Specificity in Neural Response Generation (2018, background): “Xing et al. (2017) incorporated the topic information from an external corpus into the Seq2Seq framework to guide the generation.”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “For diversity, we choose the number and proportion of distinct n-grams (Li et al., 2016; Xing et al., 2017; Ke et al., 2018) as our metric.”
Commonsense Knowledge Aware Conversation Generation with Graph Attention (2018, methodology): “These models developed various techniques to improve the content quality of generated responses, including diversity promotion [Li et al. , 2016; Shao et al. , 2017], considering additional information [Xing et al. , 2017; Mou et al. , 2016], and handling unknown words [Gu et al. , 2016 ] .”
Steering Output Style and Topic in Neural Response Generation (2017, methodology): “(Chen et al., 2017) have recently proposed using LDA for topic modeling in Sequence-To-Sequence response generation models.”
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (2018, methodology): “…on human judgments, that our approach generates more content-rich responses when compared with two competitive baselines: Maximum Mutual Information (MMI) (Li et al., 2016a), in addition to an approach that conditions on topic models as additional context in neural conversation (Xing et al., 2017).”
Response Generation by Context-aware Prototype Editing (2018, methodology): “Prior works on retrieval-based methods mainly focus on the matching model architecture for single turn conversation (Hu et al. 2014) and multi-turn conversation (Lowe et al. 2015; Zhou et al. 2016; Wu et al. 2017).”
Condition Aware and Revise Transformer for Question Answering (2020, methodology): “[31] use Twitter LDA model to get the topic of dialog and then feed topic and input representations into a joint attention module to generate a topic-related response.”
Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework (2019, background): “Various approaches (Li et al., 2016b; Xing et al., 2017; Ghazvininejad et al., 2018; Zhou et al., 2018; Liu et al., 2018; Bi et al., 2019; Tian et al., 2019; Gao et al., 2019) have been proposed for this problem.”
Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memory (2018, background): “…(Serban et al. 2017; Zhao, Lee, and Eskenazi 2018; Cao and Clark 2017; Shen et al. 2017), discourse-level variations (Zhao, Zhao, and Eske-nazi 2017), topic information (Xing et al. 2017), speaker personality (Li et al. 2016b) and knowledge base (Ghazvinine-jad et al. 2018; Zhou et al. 2018).”
Hierarchical Inductive Transfer for Continual Dialogue Learning (2022, background): “…the higher layers to improve the training efficiency.272 4 Related Work273 Continual Dialogue Learning Neural dialogue274 models (Mou et al., 2016; Xing et al., 2017; Zhao275 et al., 2017) can acquire various kinds of conversa-276 tion skills from corpora, such as characterizing per-277…”