A Simple Language Model for Task-Oriented Dialogue

By Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, Richard Socher. 2020

583 citations recorded by the source corpus.

Task-oriented dialogue is often decomposed into three tasks: understanding user input, deciding actions, and generating a response. While such decomposition might suggest a dedicated model for each sub-task, we find a simple, unified approach leads to state-of-the-art performance on the MultiWOZ dataset. SimpleTOD is a simple approach to task-oriented dialogue that uses a single causal language model trained on all sub-tasks recast as a single sequence prediction problem. This allows SimpleTOD to fully leverage transfer learning from pre-trained, open domain, causal language models such as GPT-2. SimpleTOD improves over the prior state-of-the-art by 0.49 points in joint goal accuracy for dialogue state tracking. More impressively, SimpleTOD also improves the main metrics used to evaluate action decisions and response generation in an end-to-end setting for task-oriented dialog systems: inform rate by 8.1 points, success rate by 9.7 points, and combined score by 7.2 points.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

ReAct: Synergizing Reasoning and Acting in Language Models (2022, methodology): “, 2022) and task-oriented dialogue systems like SimpleTOD (Hosseini-Asl et al., 2020) also train LMs to make decision about API calls.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “While previous works also cast SKG tasks into the text-to-text format (Hosseini-Asl et al., 2020; Shaw et al., 2021; Liu et al., 2021), their independent choices of pretrained language models (PLMs), input-output formats, and frameworks make our uniﬁcation non-trivial.”
Continual Prompt Tuning for Dialog State Tracking (2022, background): “…Gao et al., 2019) while174 others generate values directly to handle situations175 where values are missing from the context (Wu176 et al., 2019; Hosseini-Asl et al., 2020).177 Generation-based models either generate all178 (slot, value) pairs in one pass (Hosseini-Asl et al.,179 2020; Madotto…”
Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “Model In this paper, we employ a decoder-only Language Models (e.g. GPT-2), which is inline with the current state-of-the-art task-oriented dialogue models Peng et al. (2020); Hosseini-Asl et al. (2020).”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “The goal of Natural Language Generation (NLG) is to produce fuent text in many domains, such as machine translation [21], summarization [34], dialogue [30], style transfer [18], and programming code [13].”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Huang et al. (2020a) argued that responses should not be simply evaluated based on their surface-level features, and instead the topic-level features were more essential.”
SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, background): “We are aware of a concurrent work by (Hosseini-Asl et al., 2020) following this line of research.”
Language Models are Few-shot Multilingual Learners (2021, methodology): “…have been used for various sequence generation tasks, such as summarization (Raffel et al., 2020), conversational agents (Lin et al., 2020b,a; Madotto et al., 2020a; Wu and Xiong, 2020; Hosseini-Asl et al., 2020; Lin et al., 2021b), and knowledge grounding (Chen et al., 2020; Zhao et al., 2020).”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “(Budzianowski and Vuli´c, 2019; Hosseini-Asl et al., 2020; Ham et al., 2020; Lin et al., 2020; Yang et al., 2021) utilized pretrained language models such as GPT-2 (Rad-ford et al., 2019) to perform TOD tasks such as language generation or act prediction.”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, methodology): “…2019; Qin et al., 2019, 2020; Raghu et al., 2019; Haihong et al., 2019; He et al., 2020) and hybrid (Shu et al., 2018; Lei et al., 2018; Zhang et al., 2019a; Mehri et al., 2019; Chen et al., 2019; Peng et al., 2020a; Ham et al., 2020; Hosseini-Asl et al., 2020; Le et al., 2020; Lin et al., 2020).”
A Conversational Paradigm for Program Synthesis (2022, unknown): “A similar approach has also been studied in the area of task-oriented dialogues (Budzianowski and Vuli´c, 2019; Hosseini-Asl et al., 2020).”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Besides, more and more studies proposed to jointly learn these four modules based on a shared PLM. Budzianowski and Vulic [18] and Hosseini-Hosseini-Asl et al. [89] generated the dialogue state, system action and final response successively, based on the original dialogue history.”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, background): “1), following the common evaluation protocol in task-oriented dialogue (Hosseini-Asl et al., 2020; Andreas et al., 2020).”
Dialogue State Tracking with a Language Model using Schema-Driven Prompting (2021, methodology): “Among them, the autoregressive model (Peng et al., 2020a; Hosseini-Asl et al., 2020) uses a uni-directional encoder whereas the sequence-to-sequence model (Lin et al.”
Structured Prediction as Translation between Augmented Natural Languages (2021, methodology): “Other sequence-to-sequence approaches solve speciﬁc structured prediction tasks by generating the desired output directly: see for example WDec (Nayak & Ng, 2020) for entity and relation extraction, and SimpleTOD (Hosseini-Asl et al., 2020) and SOLOIST (Peng et al., 2020) for dialogue state tracking”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “…Methods We compare our method with: (i) TRADE (Wu et al., 2019): It is a transferable multi-domain generative dialogue state tracking model, (ii) SimpleTOD (Hosseini-Asl et al., 2020): This is a state-of-the-art task-speciﬁc model for dialogue state tracking based on GPT-2 (Rad-ford et al., 2019b).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “By enabling agents to function as communicators themselves, they become capable of solving complex tasks [113, 85, 72, 3, 30, 111, 79, 41, 28, 102, 80, 106, 35, 49, 2, 51, 1, 55, 50, 65, 92].”