DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation

By Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, William B. Dolan. 2019

1,737 citations recorded by the source corpus.

We present a large, tunable neural conversational response generation model, DIALOGPT (dialogue generative pre-trained transformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human both in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response generation and the development of more intelligent open-domain dialogue systems.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “The pre-trained models are usually used as the initialization for fine-tuning downstream tasks such as controllable language modeling (Shirish Keskar et al., 2019), summarization (Song et al., 2019; Liu and Lapata, 2019) and dialogue generation (Zhang et al., 2019).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “, dialogue agents (Zhang et al., 2019)) from a fairness perspective.”
MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers (2021, background): “2020) and DIALOGPT (Zhang et al. 2020d) are two comprehensive systems for cross-domain text-to-SQL and conversational response generation tasks, respectively.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “However, recent work by Fabbri et al. (2020) demonstrates that this may not hold in summarization, where the automated metric BERTScore (Zhang et al., 2020b) does not improve upon the correlation of ROUGE.”
Towards a Human-like Open-Domain Chatbot (2020, background): “End-to-end neural network approaches (Shang et al., 2015; Vinyals and Le, 2015; Sor-doni et al., 2015; Serban et al., 2016; Zhang et al., 2019), on the other hand, offer the simplicity of a single learned model.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “We use the 345M-parameter medium DialoGPT model (Zhang et al., 2020), which consists of a model with GPT-2 architecture trained on Reddit comment chains in order to expose it to dialogue, to measure bias in generations given”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “For comparison, we also measure bias on the DialoGPT dialogue model (Zhang et al., 2020), which was trained on Reddit comment chains.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “With respect to the multi-turn safety classiﬁer, results were mixed across models, with the majority of both DialoGPT and GPT-2’s responses ﬂagged as offensive.”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “10 indicate that BST 2.7B is signiﬁcantly more engaging than Di-aloGPT (both variants), and pushift.io Reddit 2.7B.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Empirical evidence suggests that data with a very large range of domains (e.g., social media data) provides the best basis for fine-tuning (Zhang et al., 2019; Shuster et al., 2020; Adiwardana et al., 2020; Roller et al., 2020).”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “BlenderBot 1 was already found to be superior to several other chatbots, in particular Meena (Adiwardana et al., 2020) and DialoGPT (Zhang et al., 2020), and we do not evaluate those here.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “For example, Zhang et al. (2020a) trains a “side” network that is fused with the pretrained model via summation; adapter-tuning inserts task-specific layers (adapters) between each layer of the pretrained LM (Houlsby et al., 2019; Lin et al., 2020; Rebuffi et al., 2017; Pfeiffer et al., 2020).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “In 2019, the 762M parameter DialoGPT model was released (Zhang et al., 2020), and in 2020 the 3B parameter”
A Simple Language Model for Task-Oriented Dialogue (2020, methodology): “[61] built on GPT-2 by further pre-training it on Reddit data for open-domain response generation.”
Survey of Hallucination in Natural Language Generation (2022, background): “The PersonaChat datasets [31, 228] have accelerated research into persona consistency [68, 91, 126, 208, 219, 224, 232].”