XCMRC: Evaluating Cross-lingual Machine Reading Comprehension
By Pengyuan Liu, Yuning Deng, Chenghao Zhu, Han Hu. 2019
9 citations recorded by the source corpus.
We present XCMRC, the first public cross-lingual language understanding (XLU) benchmark which aims to test machines on their cross-lingual reading comprehension ability. To be specific, XCMRC is a Cross-lingual Cloze-style Machine Reading Comprehension task which requires the reader to fill in a missing word (we additionally provide ten noun candidates) in a sentence written in target language (English / Chinese) by reading a given passage written in source language (Chinese / English). Chinese and English are rich-resource language pairs, in order to study low-resource cross-lingual machine reading comprehension (XMRC), besides defining the common XCMRC task which has no restrictions on use of external language resources, we also define the pseudo low-resource XCMRC task by limiting the language resources to be used. In addition, we provide two baselines for common XCMRC task and two for pseudo XCMRC task respectively. We also provide an upper bound baseline for both tasks. We found that for common XCMRC task, translation-based method and multilingual sentence encoder-based method can obtain reasonable performance but still have much room for improvement. As for pseudo low-resource XCMRC task, due to strict restrictions on the use of language resources, our two approaches are far below the upper bound so there are many challenges ahead.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Eunsol Choi
- Luke S. Zettlemoyer
- Tom Kwiatkowski
- Percy Liang
- Hannaneh Hajishirzi
- Noah A. Smith
- Jonathan Berant
- Sebastian Ruder
In Natural Language Processing
- Eunsol Choi
- Luke S. Zettlemoyer
- Tom Kwiatkowski
- Percy Liang
- Michael Collins
- Christopher D. Manning
- Greg Durrett
- Jonathan Berant
In Evaluation & Benchmarks
- Eunsol Choi
- Shayne Longpre
- Tom Kwiatkowski
- Michael Collins
- J. Clark
- Joachim Daiber
- Yi Lu
- Dan Garrette
Citations in the corpus
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “For example, in XQA (Liu et al., 2019a) and XCMRC (Liu et al., 2019b), statements phrased syntactically as questions ( Did you know that is the largest stringray?”
- MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering (2020, background): “Similarly, XCMRC (Liu et al., 2019b) and BiPar (Jing et al., 2019) present parallel, cross-lingual QA dataset be-tween English and Chinese.”