OCNLI: Original Chinese Natural Language Inference

By Hai Hu, Kyle Richardson, Lu Li, Sandra Kübler, Larry Moss. 2020

145 citations recorded by the source corpus.

Despite the tremendous recent progress on natural language inference (NLI), driven largely by large-scale investment in new datasets (e.g.,SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world’s languages. In this paper, we present the first large-scale NLI dataset (consisting of ~56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers specializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance (~12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chinese NLU. To the best of our knowledge, this is the first human-elicited MNLI-style corpus for a non-English language.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “This was recently demonstrated at a small scale by Hu et al. (2020) with OCNLI: They show that it is possible to signiﬁcantly improve data quality issues by making small interventions during the crowdsourcing process—like offering additional bonus payments for examples that avoid overused words and…”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “Dataset : We use TouTiao News Titles Classiﬁ-cation (TNEWS), IFLYTEK app description clas-siﬁcation (IFLYTEK), and Original Chinese NLI (OCNLI) as our benchmark datasets for text clas-siﬁcation (Xu et al., 2020; Hu et al., 2020).”
UnNatural Language Inference (2020, methodology): “All models are trained on OCNLI corpus (Hu et al., 2020a).”
FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (2021, unknown): “0 on EPRSTMT, OCNLI, BUSTM and CSL tasks is signiﬁcantly worse than that of RoBERTa, and the effect on CSLDCP, TNEWS and IFLYTEK tasks is obviously better than RoBERTa.”
bert2BERT: Towards Reusable Pretrained Language Models (2021, methodology): “For the evaluation, we use the dataset of the original Chinese natural language inference task (OCNLI) (Hu et al., 2020).”
Qwen Technical Report (2023, background): “• OCNLI (Hu et al., 2020) This is an NLI dataset focusing on Chinese.”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “They also show similar effects in non-English datasets (e.g. the Chinese OCNLI corpus; Hu et al., 2020), and posit overly high model confidence to be one of the factors driving this issue.”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, background): “4 Inspired by Hu et al. (2020), we ask workers to write two questions per passage with four answer choices each.”