A Span-Extraction Dataset for Chinese Machine Reading Comprehension

By Yiming Cui, Ting-Wei Liu, Li Xiao, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, Guoping Hu. 2019

214 citations recorded by the source corpus.

Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, the existing reading comprehension datasets are mostly in English. In this paper, we introduce a Span-Extraction dataset for Chinese machine reading comprehension to add language diversities in this area. The dataset is composed by near 20,000 real questions annotated on Wikipedia paragraphs by human experts. We also annotated a challenge set which contains the questions that need comprehensive understanding and multi-sentence inference throughout the context. We present several baseline systems as well as anonymous submissions for demonstrating the difficulties in this dataset. With the release of the dataset, we hosted the Second Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2018). We hope the release of the dataset could further accelerate the Chinese machine reading comprehension research. Resources are available: https://github.com/ymcui/cmrc2018

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “Regarding CMRC2018, DRCD and WebQA tasks of PanGu-α 13B, the few-shot performance surpasses zero-shot by more than 10 points, demonstrating that PanGu-α 13B has superior in-context learning ability.”
Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering (2019, methodology): “There are two additional characteristics to note about our data augmentation techniques: The most salient characteristic is that SQuAD, CMRC, and DRCD all have source answers drawn from Wikipedia (English or Chinese), while TriviaQA includes web pages as well as Wikipedia.”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, methodology): “Although more and more Chinese linguistic tasks (Liu et al., 2018; Cui et al., 2019) have been proposed, there is still a need for a standard evaluation pipeline and an evaluation benchmark with a set of diverse and difficult language understanding tasks.”
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (2019, methodology): “For MRC task, CMRC 2018 [27], DRCD [28], DuReader [29] are used as test datasets.”
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, background): “The average scores of Yuan outperform Pangu-α and Ernie 3.0 by a large margin on Close-book QA[24] and Span Extraction reading comprehension[25], which proves the excellent zero-shot generation capacity of Yuan 245B.”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “Although there are some previous works providing powerful Chinese pre-trained language models (Cui et al., 2020; Xu et al., 2020; Wei et al., 2019; Sun et al., 2019; Cui et al., 2019a), their capabilities are limited due to the model size.”
Distant Supervision for Multi-Stage Fine-Tuning in Retrieval-Based Question Answering (2020, result): “We note the same phenomenon in Chinese (for CMRC, bottom).”
A Comprehensive Overview of Large Language Models (2023, unknown): “Traditionally, these architectures perform well for different tasks, for example, encoder-only for NLU tasks, decoder-only for NLG, and encoder-decoder for sequence2sequence modeling.”
TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing (2020, methodology): “For Chinese datasets, we use XNLI (Conneau et al., 2018), LCQMC (Liu et al., 2018), CMRC 2018 (Cui et al., 2019b) and DRCD (Shao et al., 2018).”
Don't Make Your LLM an Evaluation Benchmark Cheater (2023, methodology): “We use chain-of-thought prompting and reuse the prompts provided by Wei et al. (2022) for evaluation and report the accuracy of LLMs. • Reading Comprehension Tasks : we select three English datasets RACE-Middle and RACE-High (Lai et al., 2017), CoQA (Reddy et al., 2019) and two Chinese datasets CMRC”
ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding (2020, background): “We execute extensive experiments on seven Chinese language understanding tasks, including natural language inference (XNLI; Conneau et al., 2018), machine reading comprehension (CMRC2018, DRCD, DuReader; Cui et al., 2018; Shao et al., 2018; He et al., 2017), named entity recognition (MSRA-NER; Levow”
Cross-Lingual Machine Reading Comprehension (2019, methodology): “We evaluate our approaches on two public Chinese span-extraction machine reading comprehension datasets: CMRC 2018 (simpliﬁed Chinese) (Cui et al., 2019) Note that, since the test and challenge sets are preserved by CMRC 2018 ofﬁcial to ensure the integrity of the evaluation process, we submitted…”
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2020, methodology): “two Chinese datasets: CMRC (Cui et al., 2018), DRCD (Shao et al.”
Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension (2019, methodology): “We have seen the construction of span-style Chinese machine reading comprehension datasets (Cui et al., 2016, 2018b,a; Shao et al., 2018), using Chinese news reports, books, and Wikipedia articles as source documents, similar to their English counterparts CNN/Daily Mail (Hermann et al., 2015),…”
Pre-Training with Whole Word Masking for Chinese BERT (2019, methodology): “inese datasets, including the ones that were also used in BERT and ERNIE. We adopt additional datasets for testing their performance in a wider range. Machine Reading Comprehension (MRC): CMRC 2018 (Cui et al., 2019), DRCD (Shao et al., 2018), CJRC (Duan et al., 2019) 5For further tests and TensorFl”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, background): “• Machine Reading Comprehension (MRC): DRCD [86], DuReader [87], Dureaderrobust [88], Dureaderchecklist, Dureaderyesno 18, C(3) [89], CMRC 2018 [90].”
BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels (2019, background): “1 for the English dataset in BiPaR, and the evaluation program 5 of CMRC2018 for the Chinese dataset in BiPaR.”