Reliable Measures for Aligning Japanese-English News Articles and Sentences

By Masso Utiyama, Hitoshi Isahara. 2003

239 citations recorded by the source corpus.

We have aligned Japanese and English news articles and sentences to make a large parallel corpus. We first used a method based on cross-language information retrieval (CLIR) to align the Japanese and English articles and then used a method based on dynamic programming (DP) matching to align the Japanese and English sentences in these articles. However, the results included many incorrect alignments. To remove these, we propose two measures (scores) that evaluate the validity of alignments. The measure for article alignment uses similarities in sentences aligned by DP matching and that for sentence alignment uses similarities in articles aligned by CLIR. They enhance each other to improve the accuracy of alignment. Using these measures, we have successfully constructed a large-scale article and sentence alignment corpus available to the public.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “Another direction of research is to use cross-lingual information retrieval, e.g. (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005; Rauf and Schwenk, 2009).”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “While traditional parallel corpus mining has relied on heavily engineered systems (Utiyama and Isahara, 2003; Etchegoyhen and Azpeitia, 2016), often based on metadata information (Resnik, 1999; Shi et al.”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, neutral): “(Utiyama and Isahara, 2003; Munteanu and Marcu, 2005) or machine translation, e.”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “Utiyama and Isahara (2003); Fukushima et al. (2006) worked on creating Japanese–English corpora.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, background): “, 2017, 2018), crosslingual document retrieval (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005), language models (Buck and Koehn, 2016), translation (AbdulRauf and Schwenk, 2009; Bouamor and Sajjad, 2018), or bag-of-words (Buck and Koehn, 2016).”
Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora (2017, background): “Previous research has shown that this bottleneck can be reduced by utilizing parallel portions found within comparable corpora (Utiyama and Isahara, 2003; Munteanu et al., 2004; Abdul-Rauf and Schwenk, 2009).”
Improving Machine Translation Performance by Exploiting Non-Parallel Corpora (2005, background): “Utiyama et al. (2003) evaluate their sentence pairs manually; they estimate that about 90% of the sentence pairs in their ﬁnal corpus are parallel.”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, neutral): “, 2009), and cross-language information retrieval (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005).”
Syntax-based Rewriting for Simultaneous Machine Translation (2015, methodology): “We evaluate our method on the Reuters JapaneseEnglish corpus of news articles (Utiyama and Isahara, 2003).”
Improving machine translation via triangulation and transliteration (2014, unknown): “This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC-BY-ND. to overcome sparsity (Utiyama and Isahara, 2003; Resnik and Smith, 2003).”
A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation (2007, background): “Utiyama and Isahara (2003) extract Japanese-English parallel sentences from a noisy-parallel corpus.”
A Survey of Domain Adaptation for Neural Machine Translation (2018, background): “ii) When there are not enough parallel corpora, there are also studies that generate pseudo-parallel sentences using information retrieval (Utiyama and Isahara, 2003), self-enhancing (Lambert et al.”
On the Use of Comparable Corpora to Improve SMT performance (2009, background): “Bold parts are the extra tails at the end of the sentences which we automatically removed. include (Utiyama and Isahara, 2003), who use cross-language information retrieval techniques and dynamic programming to extract sentences from an English-Japanese comparable corpus.”