A Corpus for Multilingual Document Classification in Eight Languages

By Holger Schwenk, Xian Li. 2018

150 citations recorded by the source corpus.

Cross-lingual document classification aims at training a document classifier on resources in one language and transferring it to a different language without any additional resources. Several approaches have been proposed in the literature and the current best practice is to evaluate them on a subset of the Reuters Corpus Volume 2. However, this subset covers only few languages (English, German, French and Spanish) and almost all published works focus on the the transfer between English and German. In addition, we have observed that the class prior distributions differ significantly between the languages. We argue that this complicates the evaluation of the multilinguality. In this paper, we propose a new subset of the Reuters corpus with balanced class priors for eight languages. By adding Italian, Russian, Japanese and Chinese, we cover languages which are very different with respect to syntax, morphology, etc. We provide strong baselines for all language transfer directions using multilingual word and sentence embeddings respectively. Our goal is to offer a freely available framework to evaluate cross-lingual document classification, and we hope to foster by these means, research in this important area.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “While XLU shows promising results for tasks such as cross-lingual document classification (Klementiev et al., 2012; Schwenk and Li, 2018), there are very few, if any, XLU benchmarks for more difficult language understanding tasks like natural language inference.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, methodology): “Given recent progress in cross-lingual tasks such as document classiﬁcation (Lewis et al., 2004; Klementiev et al., 2012; Schwenk and Li, 2018), semantic role labelling (Akbik et al., 2015) and NLI (Conneau et al., 2018), we argue that while multilingual QA training data might be useful but not…”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “Evaluation on a single task is problematic as past work has noted potential issues with standard datasets: MLDoc (Schwenk & Li, 2018) can be solved by matching keywords (Artetxe et al., 2019), while MultiNLI, the dataset from which XNLI (Conneau et al., 2018b) was derived, contains superﬁcial cues…”
On the Cross-lingual Transferability of Monolingual Representations (2019, background): “We describe the models that we compare ( § 3.1), the experimental setting ( § 3.2), and the results on three classiﬁcation datasets: XNLI ( § 3.3), MLDoc ( § 3.4) and PAWS-X ( § 3.5).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, methodology): “We use MLDoc (Schwenk and Li, 2018), a balanced subset of the Reuters corpus covering 8 languages for document classification.”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “These include lexical semantic similarity (Multi-SimLex) (Vuli´c et al., 2020), document classiﬁcation (MLDoc) (Schwenk and Li, 2018), sentiment analysis (Barnes et al., 2018), and natural language inference (XNLI) (Conneau et al., 2018).”
Bridging the domain gap in cross-lingual document classification (2019, methodology): “Using such techniques, recent work has demonstrated reasonable zero-shot performance for crosslingual document classification (Schwenk & Li (2018)) and natural language inference (Conneau et al. (2018)).”
XQA: A Cross-lingual Open-domain Question Answering Dataset (2019, unknown): “Recently, XLU has been applied to many natural language processing tasks such as cross-lingual document classiﬁcation (Schwenk and Li, 2018), cross-lingual natural language inference (Conneau et al., 2018b), and machine translation (Lample et al., 2018).”
XCMRC: Evaluating Cross-lingual Machine Reading Comprehension (2019, background): “While XLU shows promising results for tasks such as cross-lingual document classification [15,25], and recently XNLI [5] is released for cross-lingual natural language inference, there is no any challenging XLU benchmarks for MRC.”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “The results are consistent across 5 different tasks (MLDoc (Schwenk and Li, 2018), XNLI (Conneau et al.”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, background): “MLDOC (Schwenk and Li, 2018) is a multilingual document classification dataset with six topics.”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, background): “, 2019), document classification (Schwenk and Li, 2018), and question an-”