JESC: Japanese-English Subtitle Corpus

By Reid Pryzant, Young-joo Chung, Dan Jurafsky, Denny Britz. 2017

73 citations recorded by the source corpus.

In this paper we describe the Japanese-English Subtitle Corpus (JESC). JESC is a large Japanese-English parallel corpus covering the underrepresented domain of conversational dialogue. It consists of more than 3.2 million examples, making it the largest freely available dataset of its kind. The corpus was assembled by crawling and aligning subtitles found on the web. The assembly process incorporates a number of novel preprocessing elements to ensure high monolingual fluency and accurate bilingual alignments. We summarize its contents and evaluate its quality using human experts and baseline machine translation (MT) systems.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “0, Autshumato (Groe-newald and du Plooy, 2010), PMIndia (Haddow and Kirefu, 2020), CVIT (Srivastava et al., 2020), Inuktitut (Hernandez and Nguyen, 2020), NLPC, JESC (Pryzant et al., 2018), KFTT (Neubig, 2011), ASPEC (Nakazawa et al., 2016), Flores (Guzmán et al., 2019).”
Automatically Neutralizing Subjective Bias in Text (2019, methodology): “Last, we discarded pairs whose length ratios were beyond the 95th percentile (Pryzant et al. 2017).”
Data augmentation using back-translation for context-aware neural machine translation (2019, background): “…the last possibility ( § 6), since there are few parallel corpora for context-aware translation; existing large-scale and high-quality parallel corpora are usually obtained by extracting reliable sentence alignments from translations by humans (Nakazawa et al., 2016; Pryzant et al., 2018).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, methodology): “For English-to-Japanese, We collect 18M sentence pairs from News Complimentary, WikiMatrix (Schwenk et al., 2021) and En-Ja subtitle corpus (Pryzant et al., 2018).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, methodology): “0 (Morishita et al., 2020), a large web-based parallel corpus, Japanese-English Subtitle Corpus (JESC) (Pryzant et al., 2017), the Kyoto Free Translation Task (KFTT) corpus (Neubig, 2011), constructed from the Kyoto-related Wikipedia articles, and TED Talks (Cettolo et al., 2012).”
Neural Machine Translation with Byte-Level Subwords (2019, methodology): “Datasets We run experiments on three bilingual corpora as well as a many-to-English multilingual dataset: • English-German (En-De): we replicate the same setting of (Vaswani et al. 2017) which uses WMT 2014 1 data (newstest13 for validation and newstest14 for testing) • Japanese-English (Ja-En): we ”