The Web as a Parallel Corpus

By Philip Resnik, Noah A. Smith. 2003

676 citations recorded by the source corpus.

Parallel corpora have become an essential resource for work in multilingual natural language processing. In this article, we report on our work using the STRAND system for mining parallel text on the World Wide Web, first reviewing the original algorithm and results and then presenting a set of significant enhancements. These enhancements include the use of supervised learning based on structural features of documents to improve classification performance, a new content-based measure of translational equivalence, and adaptation of the system to take advantage of the Internet Archive for mining parallel text from the Web on a large scale. Finally, the value of these techniques is demonstrated in the construction of a significant parallel corpus for a low-density language pair.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Filtering and Mining Parallel Data in a Joint Multilingual Space (2018, background): “There is a huge body of research on mining bi-texts, e.g. by analyzing the name of WEB pages or links (Resnik and Smith, 2003).”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, background): “It uses a threshold over the strength of this alignment to filter spurious candidates (Resnik, 1999; Resnik and Smith, 2003).”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, neutral): “(Resnik, 1999; Resnik and Smith, 2003).”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “amples of structural matching is the use of editdistance between linearized documents (Resnik and Smith, 2003) and probability of a probabilistic DOM-tree alignment model (Shi et al.”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “Initial approaches to bitext mining have relied on heavily engineered systems often based on meta-data information, e.g. (Resnik, 1999; Resnik and Smith, 2003).”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, methodology): “A different class of alignment methods rely on document structure (Resnik and Smith, 2003; Chen and Nie, 2000).”
Improving Machine Translation Performance by Exploiting Non-Parallel Corpora (2005, background): “Whether this is true for comparable corpora from other domains is an empirical question outside the scope of this article; however, both our results and those of Resnik and Smith (2003) strongly indicate that good data is available on the Web.”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, neutral): “Much of the previous work on parallel document mining has relied on using metadata, such as document titles (Yang and Li, 2002), publication dates (Munteanu and Marcu, 2005, 2006) or document structure (Chen and Nie, 2000; Resnik and Smith, 2003; Shi et al., 2006), to identify bitexts.”
UM-Corpus: A Large English-Chinese Parallel Corpus for Statistical Machine Translation (2014, methodology): “During the construction process, existing mature algorithms or methods are adopted to accelerate the building process, such as document alignment (Resnik and Smith, 2003; Patry and Langlais, 2005), sentence boundary detection (Koehn, 2005; Gillick, 1 http://nlp2ct.cis.umac.mo/um-corpus/index.html…”
Dirt Cheap Web-Scale Parallel Text from the Common Crawl (2013, methodology): “Starting from nothing other than a set of language codes, our extension of the STRAND algorithm (Resnik and Smith, 2003) identifies potentially parallel documents using cues from URLs and document content (§2).”
On the Use of Comparable Corpora to Improve SMT performance (2009, methodology): “Resnik and Smith (2003) pro-pose their STRAND web-mining based system and show that their approach is able to ﬁnd large numbers of similar document pairs.”
BUCC 2017 Shared Task: a First Attempt Toward a Deep Learning Framework for Identifying Parallel Sentences in Comparable Corpora (2017, neutral): “…of the candidate sentence pairs, traditional approaches apply candidate ﬁlters such as sentence length ratio, bilingual dictionary word overlap, word alignment conditions from SMT and information retrieval systems (Resnik and Smith, 2003; Munteanu and Marcu, 2005; Abdul-Rauf and Schwenk, 2009).”
Measuring semantic similarity between words using web search engines (2007, neutral): “Resnik and Smith [33] extracted bilingual sentences from the Web to create a parallel corpora for machine translation.”
Improving machine translation via triangulation and transliteration (2014, neutral): “This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC-BY-ND. to overcome sparsity (Utiyama and Isahara, 2003; Resnik and Smith, 2003).”