OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

By Pierre Lison, Jörg Tiedemann. 2016

940 citations recorded by the source corpus.

We present a new major release of the OpenSubtitles collection of parallel corpora. The release is compiled from a large database of movie and TV subtitles and includes a total of 1689 bitexts spanning 2.6 billion sentences across 60 languages. The release also incorporates a number of enhancements in the preprocessing and alignment of the subtitles, such as the automatic correction of OCR errors and the use of meta-data to estimate the quality of each subtitle and score subtitle pairs.

Subfields: Artificial Intelligence, Language and Linguistics, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Language and Linguistics

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “For Urdu, we used the Bible and Quran transcriptions (Tiedemann, 2012), the OpenSubtitles 2016 (Pierre and Jörg, 2016) and 2018 corpora and LDC2010T21, LDC2010T23 LDC corpora, and obtained a total of 64k parallel sentences.”
Beyond English-Centric Multilingual Machine Translation (2020, background): “Other work has identified parallel data in sources such as subtitles, such as the OpenSubtitles project (Lison and Tiedemann, 2016).”
Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “n document context. As well as Europarl, Ted talk subtitles6 (Tiedemann, 2012) were included for both training and evaluation. Also available for the translation ofextended speech is Open Subtitles7 (Lison and Tiedemann, 2016; Muller et al., 2018), which contains utterance pairs which are aligned an”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “It consists of data from 22 data sources, coarsely broken down into 5 categories: • Academic Writing: Pubmed Abstracts and PubMed Central, arXiv, FreeLaw,5 USPTO Backgrounds,6 PhilPapers,7 NIH Exporter8 • Web-scrapes and Internet Resources: CommonCrawl, OpenWebText2, StackExchange,9 Wikipedia (Engli”
Multi-Dimensional Gender Bias Classification (2020, background): “, 2018), OpenSubtitles (dialogue from movies) (Lison and Tiedemann, 2016), and LIGHT (chit-chat fantasy dialogue) (Urbanek et al.”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, background): “There are several projects relying on volunteers to provide translations for public texts, e.g. news commentary (Tiedemann, 2012), OpensubTitles (Lison and Tiedemann, 2016) or the TED corpus (Qi et al., 2018) Wikipedia is probably the largest free multilingual resource on the Internet.”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, neutral): “org (Lison and Tiedemann, 2016).”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, background): “They use global image features extracted from Detectron (Girshick et al., 2018), a pre-trained object detection and localisation neural network, and two additional training corpora: MS-COCO (Lin et al., 2014) (an English multimodal dataset, which they extend with synthetic multilingual data) and Ope”
A Dataset for Document Grounded Conversations (2018, background): “Datasets based on movie scripts (Lison and Tiedemann, 2016; Danescu-Niculescu-Mizil and Lee, 2011a) contain artiﬁcial conversations.”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “Rafalovitch and Dale (2009); Ziemski et al. (2015) extract data from the United Nations, T¨ager (2011) from European Patents, Lison and Tiedemann (2016) from a collection of TV and movie subtitles.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, background): “There are several projects relying on volunteers to provide translations for public texts, e.g. news commentary (Tiedemann, 2012), Opensub-Titles (Lison and Tiedemann, 2016) or the TED corpus (Qi et al., 2018).”
Automatic Gender Identification and Reinflection in Arabic (2019, methodology): “Since the Arabic text we use from the OpenSubtitles Corpus (Lison and Tiedemann, 2016), a collection of translated movie subtitles, has many spelling errors of the above mentioned kinds, we evaluate MT within an orthographically normalized space (more details in Section 6).”
Evaluating Discourse Phenomena in Neural Machine Translation (2017, methodology): “Models are trained and tested on fan-produced parallel subtitles from OpenSubtitles2016 5 (Lison and Tiedemann, 2016).”
JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages (2019, background): “Moving from OPUS and Tatoeba towards greater linguistic breadth, there are several publicly available Bible datasets, most notably those by Mayer and Cysouw (2014) and Christodouloupoulos and Steedman (2015).”