Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairs

By K. Wołk, Krzysztof Marasek. 2015

53 citations recorded by the source corpus.

Abstract Parallel sentences are a relatively scarce but extremely useful resource for many applications including cross-lingual retrieval and statistical machine translation. This research explores our methodology for mining such data from previously obtained comparable corpora. The task is highly practical since non-parallel multilingual data exist in far greater quantities than parallel corpora, but parallel sentences are a much more useful resource. Here we propose a web crawling method for building subject-aligned comparable corpora from Wikipedia articles. We also introduce a method for extracting truly parallel sentences that are filtered out from noisy or just comparable sentence pairs. We describe our implementation of a specialized tool for this task as well as training and adaption of a machine translation system that supplies our filter with additional information about the similarity of comparable sentence pairs.

Subfields: Artificial Intelligence, Information Systems

Citations in the corpus

• SEF@UHH at SemEval-2017 Task 1: Unsupervised Knowledge-Free Semantic Textual Similarity via Paragraph Vector (2017, unknown): “, 2015)): Wikipedia (Wolk and Marasek, 2014), TED3, MultiUN (Eisele and Chen, 2010), EUBookshop (Skadiņš et al.”