JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages

By Zeljko Agic, Ivan Vulic. 2019

264 citations recorded by the source corpus.

Viable cross-lingual transfer critically depends on the availability of parallel texts. Shortage of such resources imposes a development and evaluation bottleneck in multilingual processing. We introduce JW300, a parallel corpus of over 300 languages with around 100 thousand parallel sentences per language pair on average. In this paper, we present the resource and showcase its utility in experiments with cross-lingual word embedding induction and multi-source part-of-speech projection.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “For example, several resources for African languages exist, including JW300 (Agić and Vulić, 2019) used in the masakhane machine translation effort (∀ et al.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “…used in the construction of EOBT includes: Europarl (Koehn, 2005), Paracrawl (Es-plà et al., 2019), TED57, Tanzil (Tiedemann, 2012), NewsCommentary, Wikimatrix (Schwenk et al., 2021), Wikititles, JW300 (Agi´c and Vuli´c, 2019), Opus100 (Zhang et al., 2020), SETimes (Tyers and Alperen, 2010), UNv1.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “These problems seem to originate in the codes used by jw.org9, which were apparently not checked in the creation of the JW300 dataset.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, methodology): “Most were trained on the JW300 corpus (Agi´c and Vuli´c, 2019).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, methodology): “…domains, like COVID-19 (Anastasopoulos et al., 2020) or religious texts (Christodouloupoulos and Steedman, 2015; Malaviya et al., 2017; Tiede-mann, 2018; Agi´c and Vuli´c, 2019); or have low quality because they are built using automatic approaches (Zhang et al., 2020; Schwenk et al., 2019, 2021).”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, methodology): “We create the monolingual corpora for HT and QU by concate-nating their respective Wikipedia dumps with their respective text from the JW300 corpus (Agi´c and Vuli´c, 2019).”
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, background): “Increasingly, small parallel datasets do exist for many languages (Mayer and Cysouw, 2014; Agi´c and Vuli´c, 2019; Artetxe et al., 2020; Ardila et al., 2020), but those resources were either produced at high cost, or are restricted to narrow domains.”
Machine Translation into Low-resource Language Varieties (2021, methodology): “We use training, development and test sets from the JW300 corpus (Agi´c and Vuli´c, 2019) containing 500K EN –”
FFR v1.1: Fon-French Neural Machine Translation (2020, background): “JW300 (JW) contains translations of Jehovah Witness sermons in over 100 languages, while BeninLangues (BL) contains vocabulary words, short expressions, small sentences, complex sentences, proverbs, as well as books of the Bible (Genesis 1 - Psalm 79).”
Towards Supervised and Unsupervised Neural Machine Translation Baselines for Nigerian Pidgin (2020, methodology): “The dataset used for the supervised was obtained from the JW300 large-scale, parallel corpus for Machine Translation (MT) by Agić & Vulić (2019).”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, unknown): “These include localization data (GNOME, KDE4, Ubuntu, Mozilla-I10n), religious text (JW300 (Agić and Vulić, 2019), Bible-eudin (Christodouloupoulos and Steedman, 2015), Tanzil), GlobalVoices, OpenSubtitles (Lison and Tiedemann, 2016), TED2020 (Reimers and Gurevych, 2020), WikiMatrix (Schwenk et al.”
A Survey of Multilingual Neural Machine Translation (2019, background): “Recently, the large-scale WikiMatrix corpus [128] and the JW300 corpus covering 300 lowresource languages [1] have been released.”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, methodology): “More recently released corpora covering a wide variety of languages (including South Asian languages) are WikiMatrix extracted from Wikipedia (Schwenk et al., 2019), and JW300 extracted from the Jehova’s Witnesses website (Agić and Vulić, 2019).”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Sources for parallel text can be the OPUS project (Tiedemann, 2012), Bible corpora (Mayer and Cysouw, 2014; Christodoulopoulos and Steedman, 2015) or the recent JW300 corpus (Agić and Vulić, 2019).”
Should All Cross-Lingual Embeddings Speak English? (2019, methodology): “We use data from the TED (Qi et al., 2018), OpenSubtitles (Lison and Tiedemann, 2016), WikiMa-trix (Schwenk et al., 2019), bible (Malaviya et al., 2017), and JW300 (Agi´c and Vuli´c, 2019) datasets.”