WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization

By Faisal Ladhak, Esin Durmus, Claire Cardie, K. McKeown. 2020

234 citations recorded by the source corpus.

We introduce WikiLingua, a large-scale, multilingual dataset for the evaluation of cross-lingual abstractive summarization systems. We extract article and summary pairs in 18 languages from WikiHow, a high quality, collaborative resource of how-to guides on a diverse set of topics written by human authors. We create gold-standard article-summary alignments across languages by aligning the images that are used to describe each how-to step in an article. As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstractive summarization methods on our dataset. We further propose a method for direct cross-lingual summarization (i.e., without requiring translation at inference time) by leveraging synthetic data and Neural Machine Translation as a pre-training step. Our method significantly outperforms the baseline approaches, while being more cost efficient during inference.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “[de/es] • WikiLingua (Ladhak et al., 2020) – Summarize step-by-step instructions from WikiHow in very concise sentences.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Additionally, only two of the 27 datasets were nonEnglish, despite much recent work that introduces multilingual summarization corpora (Giannakopoulos et al., 2015; Scialom et al., 2020; Ladhak et al., 2020; Hasan et al., 2021; Perez-Beltrachini and Lapata, 2021).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Wiki Lingua English (Ladhak et al., 2020) • Additional datasets that we assign to a miscellaneous task cluster include: 1.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “…RDF triple WikiAuto + Turk/ASSET (Jiang et al., 2020) (Xu et al., 2016) (Alva-Manchego et al., 2020) Communicate the same information as the source sentence using simpler words and grammar. en 594k Sentence WikiLingua (Ladhak et al., 2020) Produce high quality summaries of an instructional article.”
PaLM 2 Technical Report (2023, background): “• WikiLingua (Ladhak et al., 2020), which focuses on generating section headers for step-by-step instructions from WikiHow, in Arabic, English, Japanese, Korean, Russian, Thai, and Turkish.”
Scalable and Efficient MoE Training for Multitask Multilingual Models (2021, methodology): “Each row describes the ROUGE-1/ROUGE-2/ROUGE-L F1 scores respectively, as reported in (Faisal Ladhak and McKeown, 2020).”
MTG: A Benchmark Suite for Multilingual Text Generation (2021, background): “Ladhak et al. (2020) present WikiLingua, which is a large-scale, multilingual dataset for cross-lingual abstractive summarization systems.”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, methodology): “We adopt WikiLingua (Ladhak et al., 2020) as the benchmark dataset.”
Scaling Data-Constrained Language Models (2023, unknown): “[50] Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown.”
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, methodology): “Deﬁning W IKI L INGUA -0 zero-shot cross-lingual summarization: We leverage the W IKI L INGUA dataset (Ladhak et al., 2020; Gehrmann et al., 2021) to create a novel zero-shot cross-lingual summarization task, which we dub W IKI L INGUA -0 .”
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, methodology): “27 new tasks were selected in addition to porting the 13 existing ones (Gehrmann et al., 2021), and we also redesigned data splits for an existing task (WikiLingua, Ladhak et al., 2020).”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, background): “…as multilingual question answering (Lewis et al., 2020; Clark et al., 2020), nat-ural language inference (NLI) (Conneau et al., 2018), summarization (Ladhak et al., 2020; Hasan et al., 2021), and reasoning datasets (Ponti et al., 2020; Lin et al., 2021) altogether only cover around 30 languages.”
Models and Datasets for Cross-Lingual Summarisation (2022, methodology): “Recent efforts (Ladhak et al., 2020) have been directed at the cre-”
A Survey of Large Language Models (2023, unknown): “Conditional Text Generation WMT’14,16,19,20,21,22 [271–276], Flores-101 [277], DiaBLa [278], CNN/DailyMail [279], XSum [280], WikiLingua [281], OpenDialKG [282] SuperGLUE [283], MMLU [284], BIG-bench Hard [285], CLUE [286]”
Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization (2021, background): “Gigaword reference summaries have substantially less hallucinations than XSum (Kang and Hashimoto, 2020), and WikiHow summaries tend to be of a higher quality since they are written and curated by humans (Koupaee and Wang, 2018; Ladhak et al., 2020).”
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “We use the Wikilingua (Ladhak et al., 2020) dataset containing passage-summary pairs in four language pairs.”