Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

By Andrea Madotto, Zihan Liu. 2020

63 citations recorded by the source corpus.

Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation (NLG). A research challenge is to learn each module with the least amount of samples (i.e., few-shots) given the high cost related to the data collection. The most common and effective technique to solve this problem is transfer learning, where large language models, either pre-trained on text or task-specific data, are fine-tuned on the few samples. These methods require fine-tuning steps and a set of parameters for each task. Differently, language models, such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), allow few-shot learning by priming the model with few examples. In this paper, we evaluate the priming few-shot ability of language models in the NLU, DST, DP and NLG tasks. Importantly, we highlight the current limitations of this approach, and we discuss the possible implication for future work.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Andrea Madotto
3. Zhaojiang Lin
4. Genta Indra Winata
5. Zihan Liu
6. Wenliang Dai
7. Bing Liu
8. Nayeon Lee

In Large Language Models

1. Zhaojiang Lin
2. Pascale Fung
3. Dan Su
4. Zihan Liu
5. Zhou Yu
6. Tiezheng Yu
7. Yejin Bang
8. Andrea Madotto

In Prompting & Reasoning

1. Pascale Fung
2. Zhaojiang Lin
3. Zhou Yu

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, background): “Pre-training of models on a large corpus is known to result in the model memorizing knowledge in its parameters [121, 142, 158].”
• Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “…2020) end-to-end (Bordes and Weston, 2017; Eric et al., 2017a; Eric and Manning, 2017; Reddy et al., 2019; Madotto et al., 2018; Wu et al., 2019b; Madotto et al., 2020a; Neelakantan et al., 2019; Qin et al., 2019, 2020; Raghu et al., 2019; Haihong et al., 2019; He et al., 2020) and hybrid (Shu…”
• Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue StateTracking (2021, background): “Dialogue State Tracking has been of broad interest to the dialogue research community (Williams and Young, 2007; Williams et al., 2014; Heck et al., 2020; Liu et al., 2020; Wu et al., 2020; Madotto et al., 2020).”