EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training

By Yuxian Gu, Jiaxin Wen, Hao Sun, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao, Lei Liu, Xiaoyan Zhu, Jie Tang and 1 more. 2022

60 citations recorded by the source corpus.

Large-scale pre-training has shown remarkable performance in building open-domain dialogue systems. However, previous works mainly focus on showing and evaluating the conversational performance of the released dialogue model, ignoring the discussion of some key factors towards a powerful human-like chatbot, especially in Chinese scenarios. In this paper, we conduct extensive experiments to investigate these under-explored factors, including data quality control, model architecture designs, training approaches, and decoding strategies. We propose EVA2.0, a large-scale pre-trained open-domain Chinese dialogue model with 2.8 billion parameters, and will make our models and codes publicly available. Automatic and human evaluations show that EVA2.0 significantly outperforms other open-source counterparts. We also discuss the limitations of this work by presenting some failure cases and pose some future research directions on large-scale Chinese open-domain dialogue systems.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Minlie Huang
2. Fei Mi
3. Hao Sun
4. Yasheng Wang
5. Zhexin Zhang
6. Mykola Pechenizkiy
7. Jiawen Deng
8. Jiale Cheng

In Natural Language Processing

1. Minlie Huang
2. Zhexin Zhang
3. Hao Sun
4. Mykola Pechenizkiy
5. Jiawen Deng
6. Jiale Cheng
7. Yasheng Wang
8. Fei Mi

Citations in the corpus

• Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “(3) Data curation, which usually filters offensive content in the training data using classifiers or set generic responses to the toxic contexts (Zhou et al., 2021; Gu et al., 2022; Mi et al., 2022).”
• Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, unknown): “…2018; Dinan et al., 2021), bias (Henderson et al., 2018; Liu et al., 2020; Barikeri et al., 2021; Lee et al., 2019), privacy (Weidinger et al., 2021), sensitive topics (Xu et al., 2020; Sun et al., 2022), and moral considerations (Ziems et al., 2022; Kim et al., 2022) draw increasing attention.”
• CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023, methodology): “Many previous works are mainly focused on English, but there also are some works (Wang et al., 2020; Gu et al., 2022) that proposed pretrained dialogue generation model for Chinese.”