SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model

By Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Lidén, Jianfeng Gao. 2020

126 citations recorded by the source corpus.

This paper presents a new method SOLOIST, which uses transfer learning to efficiently build task-oriented dialog systems at scale. We parameterize a dialog system using a Transformer-based auto-regressive language model, which subsumes different dialog modules (e.g., state tracker, dialog policy, response generator) into a single neural model. We pre-train, on large heterogeneous dialog corpora, a large-scale Transformer model which can generate dialog responses grounded in user goals and real-world knowledge for task completion. The pre-trained model can be efficiently adapted to accomplish a new dialog task with a handful of task-specific dialogs via machine teaching. Our experiments demonstrate that (i) SOLOIST creates new state-of-the-art results on two well-known benchmarks, CamRest and MultiWOZ, (ii) in the few-shot learning setting, the dialog systems developed by SOLOIST significantly outperform those developed by existing methods, and (iii) the use of machine teaching substantially reduces the labeling cost. We will release our code and pre-trained models for reproducible research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “[36] proposed a GPT2-based model for end-to-end training (SOLOIST) by removing actions from the input sequence, and used token-type embedding for user and system responses as well.”
Few-shot Natural Language Generation for Task-Oriented Dialog (2020, background): “Pre-trained language models (PLMs) have substantially advanced the stateof-the-art across a variety of natural language processing (NLP) tasks (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Keskar et al., 2019; Raffel et al., 2019; Peng et al., 2020).”
Dialogue State Tracking with a Language Model using Schema-Driven Prompting (2021, methodology): “Peng et al. (2020a) also leverages other dialogue corpora to finetune the language model.”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, background): “Following Peng et al. (2020b); Kale and Rastogi (2020b), the “Naive” canonical representation A = ∑A i=1 ai(si = vi) is the set of actions produced by the system, where A is the total number of actions for this turn.”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, background): “…for tasks such as information extraction (Ritter et al., 2012), information retrieval (Guo et al., 2009), question answering (Mollá et al., 2006), task-oriented dialogues (Peng et al., 2020a; Gao et al., 2019) and other language understanding applications (Nadeau and Sekine, 2007; Shaalan, 2014).”
Structured Prediction as Translation between Augmented Natural Languages (2021, background): “Other sequence-to-sequence approaches solve speciﬁc structured prediction tasks by generating the desired output directly: see for example WDec (Nayak & Ng, 2020) for entity and relation extraction, and SimpleTOD (Hosseini-Asl et al., 2020) and SOLOIST (Peng et al., 2020) for dialogue state tracking”
MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems (2020, methodology): “In future work, we plan to explore task-oriented dialogues domain-adaptive pre-training methods (Wu et al., 2020; Peng et al., 2020) to enhance our language model backbones, and extend the framework for mixed chit-chat and task-oriented dialogue agents (Madotto et al., 2020a).”
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2020, background): “Recently, there has been growing interest in the so-called end-to-end approach to task-based dialog systems (Peng et al., 2020; Hosseini-Asl et al., 2020; Lin et al., 2020; Wen et al., 2017; Bordes et al., 2016) due to its relatively simple and scalable architecture, and promising results in chatbot”
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling (2021, background): “Recently, there has been growing interest in developing deep learning-based end-to-end ToD systems [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] because they can handle complex dialogue patterns with minimal hand-crafted rules.”
Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems (2020, methodology): “For the few-shot evaluation, we follow the setting of Peng et al. (2020b) and use the BLEU and slot error rate (SLR) as metrics.”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “We are aware of newly-released models (Hosseini-Asl et al., 2020; Peng et al., 2020a); however, no code was available at submission time for running the results on single domain.”