An Unsupervised System for Parallel Corpus Filtering
By Viktor Hangya, Alexander M. Fraser. 2018
11 citations recorded by the source corpus.
In this paper we describe LMU Munich’s submission for the WMT 2018 Parallel Corpus Filtering shared task which addresses the problem of cleaning noisy parallel corpora. The task of mining and cleaning parallel sentences is important for improving the quality of machine translation systems, especially for low-resource languages. We tackle this problem in a fully unsupervised fashion relying on bilingual word embeddings created without any bilingual signal. After pre-filtering noisy data we rank sentence pairs by calculating bilingual sentence-level similarities and then remove redundant data by employing monolingual similarity as well. Our unsupervised system achieved good performance during the official evaluation of the shared task, scoring only a few BLEU points behind the best systems, while not requiring any parallel training data.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Philipp Koehn
- Vishrav Chaudhary
- Francisco (Paco) Guzmán
- Huda Khayrallah
- Kenneth Heafield
- Juan Pino
- Graham Neubig
- Felix Stahlberg
In Natural Language Processing
- Philipp Koehn
- Vishrav Chaudhary
- Francisco (Paco) Guzmán
- Huda Khayrallah
- Kenneth Heafield
- Kevin Knight
- Juan Pino
- Nadir Durrani
Citations in the corpus
- Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “Other works based on sentence embeddings include Hangya and Fraser (2018) and Littell et al. (2018), as well as Schwenk et al. (2019), which mines millions of parallel sentences in 1620 language pairs from Wikipedia.”
- Findings of the WMT 2018 Shared Task on Parallel Corpus Filtering (2018, background): “, 2018) LMU Ludwig Maximilian University of Munich, Germany (Hangya and Fraser, 2018) MAJE WebInterpret, Spain (Fomicheva and González-Rubio, 2018) Microsoft Microsoft Corp.”
- Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, background): “Given crosslingual word embeddings, sentence match scores based on the difference between the average of the word embeddings (Paetzold, 2018), or, for each word in the sentence, the closest match in the corresponding sentence (Hangya and Fraser, 2018).”