On the Use of Comparable Corpora to Improve SMT performance

By Sadaf Abdul Rauf, Holger Schwenk. 2009

144 citations recorded by the source corpus.

We present a simple and effective method for extracting parallel sentences from comparable corpora. We employ a statistical machine translation (SMT) system built from small amounts of parallel texts to translate the source side of the non-parallel corpus. The target side texts are used, along with other corpora, in the language model of this SMT system. We then use information retrieval techniques and simple filters to create French/English parallel data from a comparable news corpora. We evaluate the quality of the extracted data by showing that it significantly improves the performance of an SMT systems.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “Other work expanded to use embeddings from translation systems (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018).”
Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “Another direction of research is to use cross-lingual information retrieval, e.g. (Utiyama and Isahara, 2003; Munteanu and Marcu, 2005; Rauf and Schwenk, 2009).”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, methodology): “In addition to that, there have been several proposals that rely on machine translation to score parallel sentence candidates using automatic metrics like BLEU and TER (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018).”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, background): “(Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018), typically to obtain an initial alignment that is then further filtered.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, background): “(Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018), typically to obtain an initial alignment that is then further filtered.”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, neutral): “ments (Munteanu and Marcu, 2005, 2006; Udupa et al., 2009; Do et al., 2009; Abdul-Rauf and Schwenk, 2009).”
LIUM’s SMT Machine Translation Systems for WMT 2011 (2011, methodology): “The general architecture of our parallel sentence extraction system is described in detail by Abdul-Rauf and Schwenk (2009). We first translated 91M words from French into English using our first stage SMT system.”
Large Scale Parallel Document Mining for Machine Translation (2010, background): “…have focused on rather homogeneous corpora and relied on metadata such as publication dates (Munteanu and Marcu, 2005; Munteanu and Marcu, 2006; Udupa et al., 2009; Do et al., 2009; Abdul-Rauf and Schwenk, 2009) or information about document structure (Resnik and Smith, 2003; Chen and Nie, 2000).”
Extracting Parallel Sentences from Comparable Corpora with STACC Variants (2018, methodology): “Thus, interesting results have been notably obtained with methods based on suffix trees (Munteanu and Marcu, 2002), maximum likelihood (Zhao and Vogel, 2002), binary classification (Munteanu and Marcu, 2005), cosine similarity (Fung and Cheung, 2004), reference metrics over statistical machine trans”
LIUM’s SMT Machine Translation Systems for WMT 2012 (2012, methodology): “The general architecture of our parallel sentence extraction system is described in detail by Abdul-Rauf and Schwenk (2009). We first translated 91M words from French into English using our first stage SMT system.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “A large variety of approaches has been explored like relying on document meta-information (Resnik, 1999), cross-lingual document retrieval (Munteanu and Marcu, 2005) or machine translation and information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018), just to name a few.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Early works on bitext mining used document metainformation (Resnik, 1999), cross-lingual document retrieval (Munteanu and Marcu, 2005) or information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018).”
BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora (2017, background): “While continuous vector representations of words and sentences estimated by neural language models and neural networks (Bengio et al., 2003; Collobert and Weston, 2008) have been successfully applied to a variety of natural language processing tasks, ranging from handwriting generation (Graves,…”
Set-Theoretic Alignment for Comparable Corpora (2016, methodology): “(Abdul-Rauf and Schwenk, 2009), for instance, apply the TER metric (Snover et al., 2006) on fully machine translated output to identify parallel sentences; (Sarikaya et al., 2009) use a similar approach but with BLEU (Papineni et al., 2002) as their similarity metric.”