Multi-turn Dialogue Response Generation with Autoregressive Transformer Models

By Oluwatobi Olabiyi, Erik T. Mueller. 2019

13 citations recorded by the source corpus.

Neural dialogue models, despite their successes, still suffer from lack of relevance, diversity, and in many cases coherence in their generated responses. These issues have been attributed to reasons including (1) short-range model architectures that capture limited temporal dependencies, (2) limitations of the maximum likelihood training objective, (3) the concave entropy profile of dialogue datasets resulting into short and generic responses, and (4) out-of-vocabulary problem leading to generation of a large number of $<$UNK$>$ tokens. Autoregressive transformer models such as GPT-2, although trained with the maximum likelihood objective, do not suffer from the out-of-vocabulary problem and have demonstrated an excellent ability to capture long-range structures in language modeling tasks. In this paper, we examine the use of autoregressive transformer models for multi-turn dialogue response generation. In our experiments, we employ small and medium GPT-2 models (with publicly available pretrained language model parameters) on the open-domain Movie Triples dataset and the closed-domain Ubuntu Dialogue dataset. The models (with and without pretraining) achieve significant improvements over the baselines for multi-turn dialogue response generation. They also produce state-of-the-art performance on the two datasets based on several metrics, including BLEU, ROGUE, and distinct n-gram.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Jason Aaron Edward Weston
4. Yejin Choi
5. Percy Liang
6. Luke S. Zettlemoyer
7. Zhe Gan
8. Graham Neubig

In Large Language Models

1. Jianfeng Gao
2. Yen-Chun Chen
3. Jingjing Liu
4. Yejin Choi
5. Minlie Huang
6. Yizhe Zhang
7. Jeff Wu
8. Siqi Sun

In Natural Language Processing

1. Jianfeng Gao
2. Michel Galley
3. Chris Brockett
4. Jason Aaron Edward Weston
5. William B. Dolan
6. Minlie Huang
7. Yizhe Zhang
8. Jingjing Liu

Citations in the corpus

• Release Strategies and the Social Impacts of Language Models (2019, background): “Entertainment Gaming [64] Chatbots [66; 47; 9] Health Medical Question-Answering systems4[30]”
• DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation (2019, background): “DLGnet (Olabiyi and Mueller, 2019) is a large transformer model trained on dialogue dataset and achieves good performance in multi-turn dialogue generation.”