Parallel corpora for medium density languages

By D. Varga, Péter Halácsy, András Kornai, Nagy Viktor, Nagy Laszlo, N. László, Tron Viktor. 2007

520 citations recorded by the source corpus.

The choice of natural language technology appropriate for a given language is greatly impacted by density (availability of digitally stored material). More than half of the world speaks medium density languages, yet many of the methods appropriate for high or low density languages yield suboptimal results when applied to the medium density case. In this paper we describe a general methodology for rapidly collecting, building, and aligning parallel corpora for medium density languages, illustrating our main points on the case of Hungarian, Romanian, and Slovenian. We also describe and evaluate the hybrid sentence alignment method we are using.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Parallel Corpus Filtering via Pre-trained Language Models (2020, methodology): “With ﬁnetuning multilingual BERT on sentences pairs aligned by Hunalign, the unsupervised acceptability already achieves comparable performance to Chaudhary et al. (2019) which use massive public parallel data.”
Parallel Data, Tools and Interfaces in OPUS (2012, methodology): “A ﬁ-nal alternative provided for the OpenSubtitles2011 corpus is an alignment based on hunalign (Varga et al., 2005): http://opus.lingﬁl.uu.se/OpenSubtitles2011/hunalign/ .”
The United Nations Parallel Corpus v1.0 (2016, methodology): “The algorithm and rules have been extracted from Psi-Toolkit (Grali´nski et al., 2012). domly select a subset of 10,000 document pairs and align them using Hunalign (Varga et al., 2005), selecting only 1-1 alignments that are themselves surrounded by 1-1 alignments.”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, methodology): “The sentence aligners are: Hunalign (Varga et al., 2005) is a widely used tool that relies on a bilingual dictionary that we generated from the Europarl corpus or other available parallel corpora.”
Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering (2018, methodology): “The Hunalign scores were obtained from the sentence aligner (Varga et al., 2005).”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, methodology): “We used the processing pipeline of the Paracrawl project to create the data, using the clean parallel data to train underlying models such as the dictionary used by Hunalign (Varga et al., 2007) and a statistical translation model used by the document aligner.”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, methodology): “org/zh/foo), and processed with BeautifulSoup, but before using Hunalign (Varga et al., 2005) to extract parallel sentence pairs.”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, methodology): “We experimented with two different aligners; hunalign (Varga et al., 2005) and Vecalign (Thompson and Koehn, 2019).”
Abu-MaTran at WMT 2015 Translation Task: Morphological Segmentation and Web Crawling (2015, methodology): “Therefore, the only modules of B ITEXTOR used for this task are those performing document and segment alignment, relying on HUNALIGN 3 (Varga et al., 2005) and an English– Finnish bilingual lexicon.”
Using LSTM to Translate French to Senegalese Local Languages: Wolof as a Case Study (2020, methodology): “Three widely used open-source tools were benchmarked: hunalign (Varga et al., 2007), yasa (Lamraoui & Langlais, 2013) and champollion (Ma, 2006).”
Findings of the WMT 2016 Bilingual Document Alignment Shared Task (2016, methodology): “Popular tools are Hunalign4 (Varga et al., 2005), Gargantua5 (Braune and Fraser, 2010), Bilingual Sentence Aligner (Moore, 2002) Bleualign6 (Sennrich and Volk, 2010), and Champollion7 (Ma, 2006).”
Multilingual Extractive Reading Comprehension by Runtime Machine Translation (2018, methodology): “We employed the hunalign sentence aligner 13 , which aligns bilingual text on the sentence level, based on the sentences’ length and dictionary-based translations (Varga et al., 2007).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, methodology): “To determine this, we split both source paragraphs and translations into sentence, and aligned them using hunalign (Varga et al., 2005) with the bitextor dictionaries (Esplà-Gomis, 2009).”
Unsupervised Parallel Sentence Extraction with Parallel Segment Detection Helps Machine Translation (2019, methodology): “Various algorithms were proposed to measure sentence similarities, such as the Hungarian alignment (Kuhn, 1955; Varga et al., 2007) and the Word Mover’s Distance (Kusner et al., 2015).”
News from OPUS — A collection of multilingual parallel corpora with tools and interfaces (2009, methodology): “We used ‘hunalign’ (Varga et al. 2007) with the ‘realign’ feature for this purpose which seems to produce very reliable results according to our experience.”
UM-Corpus: A Large English-Chinese Parallel Corpus for Statistical Machine Translation (2014, methodology): “To accelerate the editing process, SuperAlign 4 was used in practice, which is a Windows application adapted from the HunAlign (Varga et al., 2005).”
An Unsupervised System for Parallel Corpus Filtering (2018, methodology): “Hunalign scores of the sentence pairs were released with the data.”