A Large-Scale Chinese Short-Text Conversation Dataset

By Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, Minlie Huang. 2020

151 citations recorded by the source corpus.

The advancements of neural dialogue generation models show promising results on modeling short-text conversations. However, training such models usually needs a large-scale high-quality dialogue corpus, which is hard to access. In this paper, we present a large-scale cleaned Chinese conversation dataset, LCCC, which contains a base version (6.8million dialogues) and a large version (12.0 million dialogues). The quality of our dataset is ensured by a rigorous data cleaning pipeline, which is built based on a set of rules and a classifier that is trained on manually annotated 110K dialogue pairs. We also release pre-training dialogue models which are trained on LCCC-base and LCCC-large respectively. The cleaned dataset and the pre-training models will facilitate the research of short-text conversation modeling. All the models and datasets are available at this https URL.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Minlie Huang
2. Jie Tang
3. Ming Ding
4. Xiao Liu
5. Chujie Zheng
6. Xiaoyan Zhu
7. Nan Duan
8. Nanyun Peng

In Natural Language Processing

1. Minlie Huang
2. Jie Tang
3. Nan Duan
4. Ming Ding
5. Chujie Zheng
6. Xiao Liu
7. Hao Sun
8. Daxin Jiang

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “For instance, if certain authors are known for frequently posting harmful material, removing all posts from these authors can serve as an effective strategy to discard both explicit and implicit unsafe content (Dinan et al., 2019; Wang et al., 2020; Gu et al., 2022).”
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, background): “In comparison, the human-written articles are masterpieces from Chinese poems, pieces of classic novels, news articles from Sohu News, and dialogues from LCCC-large dataset[29].”
• Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022, unknown): “• CD IAL -GPT (Wang et al., 2020) trains a dialog model with 104M parameters on a cleaned Chinese dialog dataset LCCC (12M dialog sessions).”
• EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training (2021, methodology): “We can see that EVA achieves remarkably higher specificity and SSA scores than baselines, and obtains a comparable sensibleness score to CDial-GPT.”
• CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023, methodology): “We evaluate the original model (CDial-GPT) and its debiased variant models on the test sets of the LCCC-base dataset (Wang et al., 2020).”
• WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models (2021, background): “[20] established a clean Chinese dialogue dataset LCCC.”
• On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2021, background): “Generative open-domain chatbots have attracted increasing attention with the emergence of transformer-based language models pretrained on large-scale corpora (Zhang et al., 2020; Wang et al., 2020; Adiwardana et al., 2020; Roller et al., 2020).”
• ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, methodology): “…(Wang et al., 2020) 5.96 2.71 CDialGPT LCCC − base (Wang et al., 2020) 6.48 3.08 CDialGPT2 LCCC − base (Wang et al., 2020) 5.69 2.50 CDialGPT LCCC − large (Wang et al., 2020) 6.63 3.20 ProphetNet-Dialog-Zh w/o finetuning 2.54 0.75 ProphetNet-Dialog-Zh w/ finetuning 6.78 3.05 generate fluent and…”