PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

By Siqi Bao, Huang He, Fan Wang, Hua Wu. 2019

296 citations recorded by the source corpus.

Pre-training models have been proved effective for a wide range of natural language processing tasks. Inspired by this, we propose a novel dialogue generation pre-training framework to support various kinds of conversations, including chit-chat, knowledge grounded dialogues, and conversational question answering. In this framework, we adopt flexible attention mechanisms to fully leverage the bi-directional context and the uni-directional characteristic of language generation. We also introduce discrete latent variables to tackle the inherent one-to-many mapping problem in response generation. Two reciprocal tasks of response generation and latent act recognition are designed and carried out simultaneously within a shared network. Comprehensive experiments on three publicly available datasets verify the effectiveness and superiority of the proposed framework.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Minlie Huang
3. Zhiyuan Liu
4. Yang Liu
5. Xipeng Qiu
6. Maosong Sun
7. Ji-rong Wen
8. Wayne Xin Zhao

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Lu Yuan
3. Luc Van Gool
4. Dacheng Tao
5. Chunyuan Li
6. Zhe Gan
7. Ming-Hsuan Yang
8. Xiaogang Wang

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, methodology): “These examples include Meena (Adiwardana et al., 2020), Blender (Roller et al., 2021), CDial-GPT (Wang et al., 2020e), Plato (Bao et al., 2020) and Plato-2 (Bao et al., 2021), which are trained on large-scale conversation data, commonly with the seq2seq framework.”
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, methodology): “For English open-domain dialog generation, we show the results in Table 5 and Table 6, compared with strong new proposed PLATO (Bao et al., 2020), we see that ProphetNet-Dialog achieves performance improvements.”
• CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023, background): “The success of the pretrained dialogue models benefits from the increasing quantity and quality of real corpora (Gu et al., 2022; Zhang et al., 2020; Radford et al., 2018; Bao et al., 2020).”
• Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Dialogue System Open-Domain Dialogue System PersonaChat [6], DailyDialogue [6], DSTC7-AVSD [6] Perplexity BLEU†, Distinct Task-Oriented Dialogue System MultiWOZ [15] BLEU, Inform ‡, Success‡”
• i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, background): “…text summarization (Liu and Lapata, 2019; Yang et al., 2020; Lewis et al., 2020; Zhang et al., 2020; He et al., 2022), dialogue generation (Bao et al., 2020; Zhao et al., 2020; OpenAI, 2022), question answering (Rajpurkar et al., 2016; Yang et al., 2018), machine translation (Wu et al.,…”
• SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, background): “Plato (Bao et al., 2019) is a pre-trained discrete latent variable model for response generation.”
• A Comprehensive Assessment of Dialog Evaluation Metrics (2021, methodology): “GRADE (Huang et al., 2020) models topic transition dynamics in dialog by constructing a graph representation of the dialog history.”
• PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning (2020, methodology): “• PLATO (Bao et al., 2020) is trained on the basis of BERT BASE using 8.3M Twitter and Reddit conversations (Cho et al., 2014; Zhou et al., 2018; Galley et al., 2019).”
• A Large-Scale Chinese Short-Text Conversation Dataset (2020, background): “Recently, pre-training models have been applied to dialogue generation tasks and achieved state-of-the art results [38,3,1].”