TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

By Chien-Sheng Wu, Steven C. H. Hoi, Richard Socher, Caiming Xiong. 2020

350 citations recorded by the source corpus.

The use of pre-trained language models has emerged as a promising direction for improving dialogue systems. However, the underlying difference of linguistic patterns between conversational data and general text makes the existing pre-trained language models not as effective as they have been shown to be. Recently, there are some pre-training approaches based on open-domain dialogues, leveraging large-scale social media data such as Twitter or Reddit. Pre-training for task-oriented dialogues, on the other hand, is rarely discussed because of the long-standing and crucial data scarcity problem. In this work, we combine nine English-based, human-human, multi-turn and publicly available task-oriented dialogue datasets to conduct language model pre-training. The experimental results show that our pre-trained task-oriented dialogue BERT (ToD-BERT) surpasses BERT and other strong baselines in four downstream task-oriented dialogue applications, including intention detection, dialogue state tracking, dialogue act prediction, and response selection. Moreover, in the simulated limited data experiments, we show that ToD-BERT has stronger few-shot capacity that can mitigate the data scarcity problem in task-oriented dialogues.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “[55] proposed a response selection model by pre-training BERT model on multiple task-oriented corpora.”
Dialogue State Tracking with a Language Model using Schema-Driven Prompting (2021, background): “, 2020) and task-oriented dialogue systems (Mehri et al., 2019; Wu et al., 2020; Henderson et al., 2020; Peng et al., 2020b).”
Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking (2019, methodology): “In addition, we compare with ToD-BERT (Wu et al., 2020), which is a large pre-trained model based on several task-oriented dialogue datasets, and it also separately encodes dialog context and domain-slot pairs.”
Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020, methodology): “Actually, the OOS recall scores in Larson et al. (2019) and Wu et al. (2020) are much lower than those with our RoBERTa clas-siﬁer, and the trade-off with respect to the tuning process was not discussed.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “More recently, Wu et al. (2020a) pretrained Task-Oriented Dialogue BERT (TOD-BERT) and significantly improved the accuracy in the intent detection sub-task.”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “To address these issues, future work can take inspiration from work related to negative task interference (Wang et al., 2020a; Larson and Leach, 2022), transferability (Vu et al., 2020; Wu et al., 2020b; Xing et al., 2022) and lifelong learning (Wang et al., 2020b).”
SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, methodology): “Since the the special token (Wu et al., 2020a) to predict whether the items of the sequence are matched ( y = 1 ) or mismatched ( y = 0 ): We consider three types of negative samples x (cid:48) , each of which is chosen with probability 1/3: ( i ) negative belief , where only the belief item is…”
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2020, background): “Over the past few years the NLP community has responded to the lack of dialog data with larger, publicly released task-oriented datasets spanning multiple domains (Wu et al., 2020; Budzianowski and Vulić, 2019).”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, methodology): “Wu et al. (2020) showed the effectiveness of BERT for several few-shot classification tasks in ToD.”
MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems (2020, methodology): “…GCE (Nouri and Hosseini, 2018), FJST (Eric et al., 2019), HyST (Goel et al., 2019), SUMBT (Lee et al., 2019), SST (Chen et al., 2020), TOD-BERT (Wu et al., 2020), and DST-Picklist (Zhang et al., 2019a); the latter includes Neural Reading (Gao et al., 2019), TRADE (Wu et al., 2019a), COMER (Ren…”
Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue StateTracking (2021, background): “Dialogue State Tracking has been of broad interest to the dialogue research community (Williams and Young, 2007; Williams et al., 2014; Heck et al., 2020; Liu et al., 2020; Wu et al., 2020; Madotto et al., 2020).”
Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems (2020, methodology): “For the few-shot evaluation, we follow the setting of Wu et al. (2020), i.e., F1-score.”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “…systems are further classiﬁed into: modularized (Levin et al., 2000; Hori et al., 2009; Lee et al., 2009), retrieval (Henderson et al., 2019; Wu et al., 2020) end-to-end (Bordes and Weston, 2017; Eric et al., 2017a; Eric and Manning, 2017; Reddy et al., 2019; Madotto et al., 2018; Wu et…”