The United Nations Parallel Corpus v1.0

By Michal Ziemski, Marcin Junczys-Dowmunt, Bruno Pouliquen. 2016

476 citations recorded by the source corpus.

This paper describes the creation process and statistics of the official United Nations Parallel Corpus, the first parallel corpus composed from United Nations documents published by the original data creator. The parallel corpus presented consists of manually translated UN documents from the last 25 years (1990 to 2014) for the six official UN languages, Arabic, Chinese, English, French, Russian, and Spanish. The corpus is freely available for download under a liberal license. Apart from the pairwise aligned documents, a fully aligned subcorpus for the six official UN languages is distributed. We provide baseline BLEU scores of our Moses-based SMT systems trained with the full data of language pairs involving English and for all possible translation directions of the six-way subcorpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Cross-lingual Language Model Pretraining (2019, methodology): “Precisely, we use MultiUN (Ziemski et al., 2016) for French, Spanish, Russian, Arabic and Chinese, and the IIT Bombay corpus (Anoop et al., 2018) for Hindi.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “( iii ) The UNCorpus (Ziemski et al., 2016) is another multilingual dataset of parliamentary documents of United Nations, consisting of around 11 million sentences in 6 languages.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “For French, Spanish, Russian, Arabic and Chinese, we use the United Nation corpora (Ziemski et al., 2016), for German, Greek and Bulgarian, the Europarl corpora (Koehn, 2005), for Turkish, Vietnamese and Thai, the OpenSubtitles 2018 corpus (Tiedemann, 2012), and for Hindi, the IIT Bombay corpus (Ano”
Beyond English-Centric Multilingual Machine Translation (2020, background): “Most existing bitext datasets go through English, with a few domain specific exceptions such as proceedings from international organizations (Koehn, 2005; Ziemski et al., 2016).”
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, background): “…with the English pronouns. glish was most recently contested at WMT’13 (Bojar et al., 2013), which oﬀered participants 14,980,513 sentence pairs from Eu-roparl 3 (Habash et al., 2017), Common Crawl (Smith et al., 2013), the United Nations corpus 4 (Ziemski et al., 2016), and news commentary.”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “0 (Ziemski et al., 2016) for 4 epochs given the larger size of this corpus.”
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks (2017, methodology): “3 Data and Experimental Setup 3.1 Data MT We use the fully-aligned United Nations corpus for training NMT models (Ziemski et al., 2016), which includes 11 million multi-parallel sentences in six languages: Arabic (Ar), Chinese (Zh), English (En), French (Fr), Spanish (Es), and Russian (Ru).”
Equalizing Gender Bias in Neural Machine Translation with Word Embeddings Techniques (2019, background): “16,554,790 sentences from a variety of sources including United Nations (Ziemski et al., 2016), Europarl (Koehn, 2005), CommonCrawl and News available from the Workshop on Machine Translation (WMT) 1.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, neutral): “7M UNCorpus [46] 25M EUBookshop v2 [47] 5.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “We collect translation pairs from MultiUN (Ziemski et al., 2016), IIT Bom-bay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), and WikiMatrix (Schwenk et al., 2019).”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, neutral): “0 [Ziemski et al., 2016].”
A Corpus for Multilingual Document Classification in Eight Languages (2018, background): “…one trained on the Europarl corpus (Koehn, 2005) to cover the languages English, German, French, Spanish and Italian, and another one trained on the United Nations corpus (Ziemski et al., 2016) which allows to learn a joint sentence embedding for English, French, Spanish, Russian and Chinese.”
Cross-Lingual Natural Language Generation via Pre-Training (2019, methodology): “In the second stage, we use Wikipedia as the monolingual data for the DAE objective, and MultiUN (Ziemski et al., 2016) as the parallel data for the XAE objective.”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, background): “An important source of parallel texts are international organizations like the European Parliament (Koehn, 2005) or the United Nations (Ziemski et al., 2016).”
Identifying and Controlling Important Neurons in Neural Machine Translation (2018, methodology): “Data We use the United Nations (UN) parallel corpus (Ziemski et al., 2016) for all experiments.”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “Rafalovitch and Dale (2009); Ziemski et al. (2015) extract data from the United Nations, T¨ager (2011) from European Patents, Lison and Tiedemann (2016) from a collection of TV and movie subtitles.”