Baselines and Test Data for Cross-Lingual Inference

By Zeljko Agic, Natalie Schluter. 2017

29 citations recorded by the source corpus.

The recent years have seen a revival of interest in textual entailment, sparked by i) the emergence of powerful deep neural network learners for natural language processing and ii) the timely development of large-scale evaluation datasets such as SNLI. Recast as natural language inference, the problem now amounts to detecting the relation between pairs of statements: they either contradict or entail one another, or they are mutually neutral. Current research in natural language inference is effectively exclusive to English. In this paper, we propose to advance the research in SNLI-style natural language inference toward multilingual evaluation. To that end, we provide test data for four major languages: Arabic, French, Spanish, and Russian. We experiment with a set of baselines. Our systems are based on cross-lingual word embeddings and machine translation. While our best system scores an average accuracy of just over 75%, we focus largely on enabling further research in multilingual inference.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “For XQuAD, the automatically translated test sets underestimate mBERT’s true performance by 3.0 F1 / 0.2 EM points, similar to the 2.6 F1 points reported by Agi´c & Schluter (2018) when translating the test data to other languages.”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, methodology): “We extract few-shot samples from the original SNLI test split, which was translated by Agi´c & Schluter (2018).”
OCNLI: Original Chinese Natural Language Inference (2020, background): “Efforts have largely focused on automatic translation of existing English resources (Mehdad et al., 2011), sometimes coupled with smaller-scale hand annotation by native speakers (Negri et al., 2011; Agi´c and Schluter, 2017).”
XCMRC: Evaluating Cross-lingual Machine Reading Comprehension (2019, background): “Agić and Schluter (2018) provided a corpus consisting of human translations for 1332 pairs of the SNLI data into Arabic, French, Russian, and Spanish [1].”