Mining the Web for Bilingual Text

By Philip Resnik. 1999

STRAND (Resnik, 1998) is a language-independent system for automatic discovery of text in parallel translation on the World Wide Web. This paper extends the preliminary STRAND results by adding automatic language identification, scaling up by orders of magnitude, and formally evaluating performance. The most recent end-product is an automatically acquired parallel corpus comprising 2491 English-French document pairs, approximately 1.5 million words per language.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, background): “These corpora are limited in size and domain, and an alternative is to mine parallel data (Resnik, 1999; Utiyama and Isahara, 2003) in large collections of monolingual data (Conneau et al., 2019; Wenzek et al., 2019).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Corpus Mining : Most NLP tasks benefit from extracting useful examples from corpora such as machine translation (MT) with bitext mining (Resnik, 1999) and argument mining.”
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings (2018, background): “Early approaches were mostly based on metadata information from web crawls (Resnik, 1999; Shi et al., 2006).”
WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia (2019, neutral): “(Resnik, 1999; Resnik and Smith, 2003).”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, background): “While the idea of mining the web for parallel data has been already pursued in the 20th century (Resnik, 1999), the most serious efforts have”
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “Initial approaches to bitext mining have relied on heavily engineered systems often based on meta-data information, e.g. (Resnik, 1999; Resnik and Smith, 2003).”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, background): “The concept of crawling and mining the web to identify sources of parallel data has been previously explored (Resnik, 1999).”
Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering (2018, background): “Although the idea of crawling the web indiscrimi-nately for parallel data goes back to the 20th century (Resnik, 1999), work in the academic community on extraction of parallel corpora from the web has so far mostly focused on large stashes of multilingual content in homogeneous form, such as the…”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, background): “One longstanding approach for extracting parallel corpora is to mine documents from the web (Resnik, 1999).”
Europarl: A Parallel Corpus for Statistical Machine Translation (2005, background): “Resnik [1999] proposes such a system, called STRAND.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “A large variety of approaches has been explored like relying on document meta-information (Resnik, 1999), cross-lingual document retrieval (Munteanu and Marcu, 2005) or machine translation and information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018), just to name a few.”
The Web as a Parallel Corpus (2003, methodology): “STRAND (Resnik 1998, 1999) is an architecture for structural translation recognition, acquiring natural data.”
Findings of the WMT 2016 Bilingual Document Alignment Shared Task (2016, background): “Although the idea of crawling the web indiscrimi-nately for parallel data goes back to the 20th century (Resnik, 1999), work in the academic community on extraction of parallel corpora from the web has so far mostly focused on large stashes of multilingual content in homogeneous form, such as the…”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “Early works on bitext mining used document metainformation (Resnik, 1999), cross-lingual document retrieval (Munteanu and Marcu, 2005) or information retrieval (Abdul-Rauf and Schwenk, 2009; Bouamor and Sajjad, 2018).”
Corpus-based Learning of Analogies and Semantic Relations (2005, methodology): “For example, Resnik (1999a) used AltaVista to ﬁnd bilingual text.”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, background): “Although the idea of crawling the web indiscrimi-nately for parallel data goes back to the 20th century (Resnik, 1999), work in the academic community on extraction of parallel corpora from the web has so far mostly focused on large stashes of multilingual content in homogeneous form, such as the…”
Neural Machine Translation: A Review (2019, background): “Unfortunately, MT training data is usually inherently noisy as it is often extracted (semi-) automatically by crawling the web [446, 447] and therefore commonly contains sentence fragments, wrong languages, misaligned sentence pairs [448], or MT output rather than genuine parallel text [449, 450].”