Parallel Corpus Filtering via Pre-trained Language Models

By Boliang Zhang, Ajay Nagesh, Kevin Knight. 2020

35 citations recorded by the source corpus.

Web-crawled data provides a good source of parallel corpora for training machine translation models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical machine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sentence parallelism by leveraging the multilingual capability of BERT and use the Generative Pre-training (GPT) language model as a domain filter to balance data domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus Filtering shared task, and on our own web-crawled Japanese-Chinese parallel corpus. Our method significantly outperforms baselines and achieves a new state-of-the-art. In an unsupervised setting, our method achieves comparable performance to the top-1 supervised method. We also evaluate on a web-crawled Japanese-Chinese parallel corpus that we make publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Graham Neubig
2. Philipp Koehn
3. Luke S. Zettlemoyer
4. Felix Stahlberg
5. Mohit Bansal
6. Noah A. Smith
7. Yejin Choi
8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

1. Luke S. Zettlemoyer
2. Felix Stahlberg
3. Philipp Koehn
4. Graham Neubig
5. Jiatao Gu
6. Kyunghyun Cho
7. Marjan Ghazvininejad
8. Mohit Bansal

In Natural Language Processing

1. Graham Neubig
2. Philipp Koehn
3. Luke S. Zettlemoyer
4. Ondrej Bojar
5. Noah A. Smith
6. Mohit Bansal
7. Kyunghyun Cho
8. Jiatao Gu

In Representation & Self-Supervised Learning

1. Daniel Khashabi
2. Luke S. Zettlemoyer
3. Marjan Ghazvininejad
4. Vishrav Chaudhary
5. Philipp Koehn
6. Jiatao Gu
7. Jungo Kasai
8. Mona T. Diab

Citations in the corpus

• Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, methodology): “Many systems instead use mined and filtered parallel data to train NMT models (Junczys-Dowmunt, 2018; Zhang et al., 2020; Koehn et al., 2019).”
• Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “We used a parallel data harvesting pipeline developed at DiDi (Zhang et al., 2020) to harvest Russian/Chinese parallel data on the Internet.”
• The Curious Case of Hallucinations in Neural Machine Translation (2021, unknown): “Corpus-Level Filtering Incorporating heuristics or filters (Junczys-Dowmunt, 2018; Zhang et al., 2020) to remove invalid source-target pairs, especially the noise patterns explored in section 4.”