Billions of Parallel Words for Free: Building and Using the EU Bookshop Corpus

By Raivis Skadins, Jörg Tiedemann, Roberts Rozis, Daiga Deksne. 2014

80 citations recorded by the source corpus.

The European Union is a great source of high quality documents with translations into several languages. Parallel corpora from its publications are frequently used in various tasks, machine translation in particular. A source that has not systematically been explored yet is the EU Bookshop ― an online service and archive of publications from various European institutions. The service contains a large body of publications in the 24 official of the EU. This paper describes our efforts in collecting those publications and converting them to a format that is useful for natural language processing in particular statistical machine translation. We report our procedure of crawling the website and various pre-processing steps that were necessary to clean up the data after the conversion from the original PDF files. Furthermore, we demonstrate the use of this dataset in training SMT models for English, French, German, Spanish, and Latvian.

Subfields: Law, Political Science and International Relations

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Law

1. Philipp Koehn
2. Huda Khayrallah

In Political Science and International Relations

1. Philipp Koehn
2. Huda Khayrallah

Citations in the corpus

• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, neutral): “7M UNCorpus [46] 25M EUBookshop v2 [47] 5.”
• Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering (2018, background): “A nice collection of the products of these efforts is the OPUS web site 1 (Skadin¸ˇs et al., 2014).”
• On the Impact of Various Types of Noise on Neural Machine Translation (2018, methodology): “Hence, we use the EU Bookstore corpus (Skadiņš et al., 2014).”
• FlauBERT: Unsupervised Language Model Pre-training for French (2019, neutral): “rawl (Buck et al., 2014) 43.4 GB 7.85 B 293.37M NewsCrawl (Li et al., 2019) 9.2 GB 1.69 B 63.05M Wikipedia (Meta, 2019) 4.2 GB 750.76M 31.00M Wikisource (Meta, 2019) 2.4 GB 458.85M 27.05M EU Bookshop(Skadins et al., 2014) 2.3 GB 389.40M 13.18M MultiUN (Eisele and Chen, 2010) 2.3 GB 384.42M 10.66M GI”
• SEF@UHH at SemEval-2017 Task 1: Unsupervised Knowledge-Free Semantic Textual Similarity via Paragraph Vector (2017, unknown): “, 2015)): Wikipedia (Wolk and Marasek, 2014), TED3, MultiUN (Eisele and Chen, 2010), EUBookshop (Skadiņš et al., 2014), SETIMES4, Tatoeba5, WMT6 and News Commentary7.”