End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2

By Dong-hyun Ham, Jeong-Gwan Lee, Youngsoo Jang, Kyungmin Kim. 2020

199 citations recorded by the source corpus.

The goal-oriented dialogue system needs to be optimized for tracking the dialogue flow and carrying out an effective conversation under various situations to meet the user goal. The traditional approach to build such a dialogue system is to take a pipelined modular architecture, where its modules are optimized individually. However, such an optimization scheme does not necessarily yield the overall performance improvement of the whole system. On the other hand, end-to-end dialogue systems with monolithic neural architecture are often trained only with input-output utterances, without taking into account the entire annotations available in the corpus. This scheme makes it difficult for goal-oriented dialogues where the system needs to integrate with external systems or to provide interpretable information about why the system generated a particular response. In this paper, we present an end-to-end neural architecture for dialogue systems that addresses both challenges above. In the human evaluation, our dialogue system achieved the success rate of 68.32%, the language understanding score of 4.149, and the response appropriateness score of 4.287, which ranked the system at the top position in the end-to-end multi-domain dialogue system task in the 8th dialogue systems technology challenge (DSTC8).

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “[17] fine-tuned GPT-2 on the MultiWOZ dataset and achieved lower performance on dialogue state tracking and end-to-end evaluation compared to the previous single-task and modularized models.”
SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, background): “However, Ham et al. (2020) directly ﬁne-tuned GPT-2 on new domains, which shows inferior performance than S OLOIST .”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “GPT-2 has been successfully exploited for various NLP and source code analysis tasks [10, 16, 28, 49].”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “(Budzianowski and Vuli´c, 2019; Hosseini-Asl et al., 2020; Ham et al., 2020; Lin et al., 2020; Yang et al., 2021) utilized pretrained language models such as GPT-2 (Rad-ford et al., 2019) to perform TOD tasks such as language generation or act prediction.”
Efficient Retrieval Augmented Generation from Unstructured Knowledge for Task-Oriented Dialog (2021, methodology): “It is used either as a model in one of the subtasks, like response generation, or as an end-to-end model for the whole task (Budzianowski and Vulić 2019; Ham et al. 2020).”
You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas (2022, unknown): “, 2019), some attempts (Wolf et al., 2019; Zhang et al., 2020; Ham et al., 2020; Shen et al., 2021; Sevegnani et al., 2021; Gu et al., 2021b) are made to build chatbots based on large generative language models (LMs).”
Dialogue State Tracking with a Language Model using Schema-Driven Prompting (2021, background): “Recently, generation-based models that are built on large-scale autoregressive pretrained language models (Ham et al., 2020; Hosseini-Asl et al., 2020; Peng et al., 2020a) achieve promising state tracking results on MultiWOZ 2.0 and 2.1 when trained on additional supervision signals or dialogue…”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Henderson et al. (2013) was the first who used a deep learning model in the DST tasks. They integrated many feature functions (e.g., SLU score, Rank score, Affirm score, etc.) as the input of a neural network, then predict the probability of each slot-value pair. Mrkšić et al. (2015) applied an RNN ”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, background): “Ham et al. (2020); Hosseini-Asl et al. (2020); Peng et al. (2020a) proposed to train GPT-2 on different sub-tasks (dialog state tracking, dialog act prediction, and response generation) as a sequence prediction problem.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Similar to segment embeddings used in BERT, dialogue state embeddings [6, 189] are used to assign each utterance, and user embeddings [6, 69] are utilized to differentiate characters involved in a conversation.”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, unknown): “…2019; Qin et al., 2019, 2020; Raghu et al., 2019; Haihong et al., 2019; He et al., 2020) and hybrid (Shu et al., 2018; Lei et al., 2018; Zhang et al., 2019a; Mehri et al., 2019; Chen et al., 2019; Peng et al., 2020a; Ham et al., 2020; Hosseini-Asl et al., 2020; Le et al., 2020; Lin et al., 2020).”