Parallel Data, Tools and Interfaces in OPUS

By Jörg Tiedemann. 2012

2,112 citations recorded by the source corpus.

This paper presents the current status of OPUS, a growing language resource of parallel corpora and related tools. The focus in OPUS is to provide freely available data sets in various formats together with basic annotation to be useful for applications in computational linguistics, translation studies and cross-linguistic corpus studies. In this paper, we report about new data sets and their features, additional annotation tools and models provided from the website and essential interfaces and on-line services included in the project.

Subfields: Computer Networks and Communications, Hardware and Architecture, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “( v ) Other smaller parallel corpora for speciﬁc domains are indexed by OPUS (Tiedemann, 2012) for various language pairs.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “, 2016), for German, Greek and Bulgarian, the Europarl corpora (Koehn, 2005), for Turkish, Vietnamese and Thai, the OpenSubtitles 2018 corpus (Tiedemann, 2012), and for Hindi, the IIT Bombay corpus (Anoop et al.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “Finally, the OPUS100 non-English directions come only with a test set, so we generate with beam size 5, length penalty 1, and use the best checkpoint.”
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “nt, whole days of European parliament, for instance. DiscoMT was introduced as a workshop in 20135 to target the translation of pronouns in document context. As well as Europarl, Ted talk subtitles6 (Tiedemann, 2012) were included for both training and evaluation. Also available for the translation ”
Toward Gender-Inclusive Coreference Resolution (2019, background): “” Even limiting to just she and he pronouns, the bias is significant: four times as many he than she in Europarl [Koehn 2005] and the Common Crawl [Smith et al. 2013], six times as many in News Commentaries [Tiedemann 2012], and fifty times as many in Hong Kong Laws corpus.”
GAIT: A Geometric Approach to Information Theory (2019, methodology): “As an example, we take an article from the News Commentary Parallel Corpus (Tiedemann, 2012), using as a measure P the normalized TF-IDF weights of each non-stopword in the article.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “…translation pairs dataset used in the construction of EOBT includes: Europarl (Koehn, 2005), Paracrawl (Es-plà et al., 2019), TED57, Tanzil (Tiedemann, 2012), NewsCommentary, Wikimatrix (Schwenk et al., 2021), Wikititles, JW300 (Agi´c and Vuli´c, 2019), Opus100 (Zhang et al., 2020),…”
Neural Approaches to Conversational AI (2018, neutral): “made available on the opensubtitles.org website, which offers captions of many commercial movies in different languages. This dataset contains about 8 billion words as of 2011, in multiple languages (Tiedemann, 2012). Ubuntu: The Ubuntu dataset (Lowe et al., 2015) has also been used extensively for ”
Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “In a supervised mode, PBSMT using the noisy and outof-domain 800,000 parallel sentences from Tiedemann (2012) achieves a BLEU score of 9.”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, background): “Large datasets have driven rapid improvement in other natural language generation tasks, such as machine translation, where data size and diversity have proven critical for modeling the alignment between source and target texts (Tiedemann, 2012).”
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (2018, neutral): “4 We collect training corpora for 93 input languages by combining the Europarl, United Nations, OpenSubtitles2018, Global Voices, Tanzil, and Tatoeba corpuses, which are all publicly available on the OPUS Web site 5 (Tiedemann, 2012).”
Semantically Equivalent Adversarial Rules for Debugging NLP models (2018, background): “, 2017) for English↔Portuguese and English↔French models, on 2 million and 1 million parallel sentences (respectively) from EuroParl, news, and other sources (Tiedemann, 2012).”
Word Translation Without Parallel Data (2017, methodology): “To show the impact of such a dictionary on machine translation, we apply it to the English-Esperanto Tatoeba corpora (Tiedemann, 2012).”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “There are two primary movie subtitle datasets: the OpenSubtitles (Tiedemann, 2012) and the SubTle Corpus (Ameixa and Coheur, 2013).”
Six Challenges for Neural Machine Translation (2017, methodology): “For the domain experiments, we use the OPUS corpus 5 (Tiedemann, 2012).”
Learning bilingual word embeddings with (almost) no bilingual data (2017, methodology): “The training and test sets were derived from a dictionary built form Europarl word alignments and available at OPUS (Tiedemann, 2012), taking 1,500 random entries uniformly distributed in 5 frequency bins as the test set and the 5,000 most frequent of the remaining word pairs as the training set.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, methodology): “4 The parallel data come from different sources, mainly from Europarl (Koehn, 2005), News Commentary (Tiedemann, 2012) and ParaCrawl.”
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles (2016, background): “In addition to increasing the global volume of the dataset by approximately 39 % compared to the previous version (Tiedemann, 2012), the release also includes several important improvements regarding how the subtitles are preprocessed and aligned with one another.”