Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems

By Paweł Budzianowski, Ivan Vulic. 2019

340 citations recorded by the source corpus.

Data scarcity is a long-standing and crucial challenge that hinders quick development of task-oriented dialogue systems across multiple domains: task-oriented dialogue models are expected to learn grammar, syntax, dialogue reasoning, decision making, and language generation from absurdly small amounts of task-specific data. In this paper, we demonstrate that recent progress in language modeling pre-training and transfer learning shows promise to overcome this problem. We propose a task-oriented dialogue model that operates solely on text input: it effectively bypasses explicit policy and language generation modules. Building on top of the TransferTransfo framework (Wolf et al., 2019) and generative model pre-training (Radford et al., 2019), we validate the approach on complex multi-domain task-oriented dialogues from the MultiWOZ dataset. Our automatic and human evaluations show that the proposed model is on par with a strong task-specific neural baseline. In the long run, our approach holds promise to mitigate the data scarcity problem, and to support the construction of more engaging and more eloquent task-oriented conversational agents.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Release Strategies and the Social Impacts of Language Models (2019, background): “Entertainment Gaming [64] Chatbots [66; 47; 9] Health Medical Question-Answering systems4[30]”
A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “Budzianowski and Vulić [6] employed GPT-2 to leverage a pre-trained language model for dialogue response generation.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, background): “Fine-tuning large pre-trained models has shown positive results in dialogue tasks (Wolf et al., 2019b; Budzianowski and Vuli´c, 2019) and other language generation tasks (Dong et al., 2019).”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Via self-supervised learning, these models acquire rich linguistic and semantic knowledge, which has been shown to inform downstream tasks in NLP [7, 21].”
PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers (2020, background): “L G ] 7 O c t 2020 ata, 2019), grammar correction (Bryant et al., 2017), entity recognition, dialogue generation (Budzianowski and Vuli´c, 2019), and more.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “the dialog scenario via encoding with dialog state embeddings[155], classifying golden and negatively sampled responses given the same dialog context[39], designing dialog-specific pre-training tasks [11, 84], and so on. These models have shown strong performance in the NeurIPS Conversational Intell”
Fine-Tuning a Transformer-Based Language Model to Avoid Generating Non-Normative Text (2020, methodology): “For example, GPT-2 has been used to generate patent claims [ Lee and Hsiang, 2019 ] and to power task-oriented conversational dialogues [Budzianowski and Vulic, 2019]; Grover can generate news articles found to be equally plausible to real articles.”
Speech-Language Pre-Training for End-to-End Spoken Language Understanding (2021, background): “, GPT), for NLP tasks like natural language understanding, natural language generation, or end-to-end task completion have been explored for dialog systems [16, 17, 18, 19].”
Few-Shot Natural Language Generation by Rewriting Templates (2020, background): “Pre-trained generative models have shown promising results for NLG in dialogue systems in low resource settings (Budzianowski and Vulic, 2019; Peng et al., 2020; Kale and Roy, 2020).”
Fast Domain Adaptation for Goal-Oriented Dialogue Using a Hybrid Generative-Retrieval Transformer (2020, methodology): “Speciﬁcally, applied to dialogue response generation, the most successful approaches made use of Transformer for chat-oriented dialogue [12] and GPT/GPT-2 for goal-oriented dialogue [13].”
Pretrained Language Models for Dialogue Generation with Multiple Input Sources (2020, methodology): “Researchers have then utilized them in dialogue generation tasks (Budzianowski and Vulić, 2019; Edunov et al., 2019; Zhang et al., 2019).”
End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2 (2020, methodology): “Parallel and independent to our work towards DSTC8 submission, Budzianowski and Vuli´c (2019) also demonstrated a neural model for goal-oriented dialogue systems by ﬁne-tuning GPT-2 on the MultiWOZ dataset.”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, methodology): “GPT2 is applied by Budzianowski and Vulic (2019); Mi et al. (2020) to train a response generation model. Ham et al. (2020); Hosseini-Asl et al. (2020); Peng et al. (2020a) proposed to train GPT-2 on different sub-tasks (dialog state tracking, dialog act prediction, and response generation) as a sequ”
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2020, background): “Over the past few years the NLP community has responded to the lack of dialog data with larger, publicly released task-oriented datasets spanning multiple domains (Wu et al., 2020; Budzianowski and Vulić, 2019).”
A Conversational Paradigm for Program Synthesis (2022, positive): “A similar approach has also been studied in the area of task-oriented dialogues (Budzianowski and Vuli´c, 2019; Hosseini-Asl et al., 2020).”
ChatGPT is not all you need. A State of the Art Review of large Generative AI models (2023, background): “Concretely, VisualGPT leverages knowledge from the pretrained language model GPT-2 [5].”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “(Budzianowski and Vuli´c, 2019; Hosseini-Asl et al., 2020; Ham et al., 2020; Lin et al., 2020; Yang et al., 2021) utilized pretrained language models such as GPT-2 (Rad-ford et al., 2019) to perform TOD tasks such as language generation or act prediction.”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “GPT-2 has been successfully exploited for various NLP and source code analysis tasks [10, 16, 28, 49].”