Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems

By Andrea Madotto, Chien-Sheng Wu, Pascale Fung. 2018

320 citations recorded by the source corpus.

End-to-end task-oriented dialog systems usually suffer from the challenge of incorporating knowledge bases. In this paper, we propose a novel yet simple end-to-end differentiable model called memory-to-sequence (Mem2Seq) to address this issue. Mem2Seq is the first neural generative model that combines the multi-hop attention over memories with the idea of pointer network. We empirically show how Mem2Seq controls each generation step, and how its multi-hop attention mechanism helps in learning correlations between memories. In addition, our model is quite general without complicated task-specific designs. As a result, we show that Mem2Seq can be trained faster and attain the state-of-the-art performance on three different task-oriented dialog datasets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “Several works used both dialogue history and knowledge bases as input and optimized neural encoder-decoder models to generate or retrieve system responses without modular supervision [13, 62, 29, 54, 56].”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “Ruiqi For the KVRET dataset, instead of the version used in our main tables, we re-run another more widely used pre-processed version (Madotto et al., 2018; Wu et al., 2019; Qin et al., 2020) on T5-base, T5-large and T5-3b.”
Neural Approaches to Conversational AI (2018, neutral): “The first is based on supervised learning, where desired system responses are first collected and then used to train multiple components of a dialogue system in order to maximize prediction accuracy (Bordes et al., 2017; Wen et al., 2017; Yang et al., 2017b; Eric et al., 2017; Madotto et al., 2018; ”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “…signiﬁcantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et al., 2018).”
Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking (2021, neutral): “Baseline Methods First, we compare TPEM with three widely used TDSs: Ptr-Unk (Eric and Manning, 2017), Mem2Seq (Madotto et al., 2018), and GLMP (Wu et al.”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “…such as machine translation (Zaremoodi et al., 2018), reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2019) and dialogue system (Madotto et al., 2018), but also some early works use text as additional information (Xie et al., 2016; An et al., 2018) or jointly train the…”
Global-to-local Memory Pointer Networks for Task-Oriented Dialogue (2019, methodology): “The hyper-parameters such as hidden size and dropout rate are tuned with grid-search over (Seo et al., 2017), End-to-end Memory Network (Bordes & Weston, 2017), Gated Memory Network (Liu & Perez, 2017), Point to Unknown Word (Gulcehre et al., 2016), and Memory-to-Sequence (Madotto et al., 2018).”
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems (2019, methodology): “, 2015), hardgated copy (Gulcehre et al., 2016; Madotto et al., 2018; Wu et al., 2019) and soft-gated copy (See et al.”
Personalizing Dialogue Agents via Meta-Learning (2019, unknown): “, 2019) and task-oriented dialogues systems (Madotto et al., 2018; Wu et al., 2019, 2017, 2018; Reddy et al., 2018).”
Dynamic Fusion Network for Multi-Domain End-to-end Task-Oriented Dialog (2020, methodology): “Follow the prior work (Eric et al., 2017; Madotto et al., 2018; Wen et al., 2018; Wu et al., 2019a; Qin et al., 2019b), we adopt the BLEU and Micro Entity F1 metrics to evaluate model performance.”
Lifelong Learning Dialogue Systems: Chatbots that Self-Learn On the Job (2020, background): “…and questionanswering systems) have an explicit knowledge base storing real-world facts [e.g., (Chicago, CityOf, USA)] (Eric and Manning 2017; Madotto, Wu, and Fung 2018; Ghazvininejad et al. 2018; Le, Dymetman, and Renders 2016; Young et al. 2018; Long et al. 2017; Zhou et al. 2018b) to…”
Multi-Level Memory for Task Oriented Dialogs (2018, unknown): “From Table 5, it can be seen that our model is able to correctly copy a signiﬁcantly larger number of entities from both, KB and context, as compared to the recent Mem2Seq model in all datasets.”
Improving Large-Scale Fact-Checking using Decomposable Attention Models and Lexical Tagging (2018, result): “Madotto et al., 2018) also have similar aspects to these works, although aiming at a different goal.”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, methodology): “Most of the implementations use either the Gold KB as input (Eric et al., 2017a; Madotto et al., 2018; Qin et al., 2019, 2020; Banerjee and Khapra, 2019; Neelakantan et al., 2019) or an intermediate API call to retrieve part of the KB ( API+KB ) (Bordes and Weston, 2017; Eric and Manning, 2017;…”
Lifelong and Interactive Learning of Factual Knowledge in Dialogues (2019, background): “Many task-oriented dialogue systems (Eric and Manning, 2017; Madotto et al., 2018) also use KBs to support information-seeking conversations.”
Contextualize Knowledge Bases with Transformer for End-to-end Task-Oriented Dialogue Systems (2020, unknown): “• Mem2Seq (Madotto et al., 2018): Mem2Seq combines the multi-hop attention mechanism in memory networks with the pointer networks to incorporate external information.”
Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations (2019, background): “…2018; Parthasarathi and Pineau, 2018; Liu et al., 2018; Moghe et al., 2018; Dinan et al., 2019; Zhong et al., 2019), task-oriented dialogue systems (Madotto et al., 2018; Wu et al., 2019; He et al., 2019) and question answering systems (Kiddon et al., 2016; Hao et al., 2017; Sun et al., 2018;…”