Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining
By Ivana Kvapilíková, Mikel Artetxe, Gorka Labaka, Eneko Agirre, Ondrej Bojar. 2020
43 citations recorded by the source corpus.
Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddings relying only on monolingual data. We first produce a synthetic parallel corpus using unsupervised machine translation, and use it to fine-tune a pretrained cross-lingual masked language model (XLM) to derive the multilingual sentence representations. The quality of the representations is evaluated on two parallel corpus mining tasks with improvements of up to 22 F1 points over vanilla XLM. In addition, we observe that a single synthetic bilingual corpus is able to improve results for other language pairs.
Subfields: Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Luke S. Zettlemoyer
- Angela Fan
- Michael Auli
- Graham Neubig
- Jason Aaron Edward Weston
- Edouard Grave
- Vishrav Chaudhary
- Armand Joulin
In Representation & Self-Supervised Learning
- Vishrav Chaudhary
- Edouard Grave
- Michael Auli
- Armand Joulin
- Hongyu Gong
- Naman Goyal
- Angela Fan
- Paul-Ambroise Duquenne
Citations in the corpus
- Beyond English-Centric Multilingual Machine Translation (2020, methodology): “Most recent methods build this similarity by comparing the embeddings from a neural network trained on multilingual data (Artetxe and Schwenk, 2018b; Chen et al., 2020; Kvapilíková et al., 2020).”
- CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2019, methodology): “In contrast, we use massively multilingual sentence embeddings trained on almost 100 languages, and then conduct margin-based mining in the multilingual embedding space (Schwenk, 2018; Artetxe and Schwenk, 2018a,b; Kvapilı́ková et al., 2020).”
- Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “The multilingual sentence representations can be learned with neural machine translation framework (España-Bonet et al., 2017; Schwenk and Douze, 2017; Artetxe and Schwenk, 2019; Kvapilíková et al., 2020).”