Explicit Alignment Objectives for Multilingual Bidirectional Encoders

By Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, Graham Neubig. 2020

65 citations recorded by the source corpus.

Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have proven impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit objective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. We conduct experiments on zero-shot cross-lingual transfer learning for different tasks including sequence tagging, sentence retrieval and sentence classification. Experimental results on the tasks in the XTREME benchmark (Hu et al., 2020) show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLM-R-large model which has 3.2x the parameters of AMBER. Our code and models are available at http://github.com/junjiehu/amber.

Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Graham Neubig
3. Li Dong
4. Ankur Bapna
5. Yu Zhang
6. Shuming Ma
7. Mohit Bansal
8. Shaohan Huang

In Natural Language Processing

1. Graham Neubig
2. Furu Wei
3. Li Dong
4. Ankur Bapna
5. Melvin Johnson
6. Orhan Firat
7. Yu Zhang
8. Luke S. Zettlemoyer

In Machine Translation

1. Zi-Yi Dou
2. Ye Jia
3. Graham Neubig
4. Ankur Bapna
5. Colin Cherry
6. Li Dong
7. Melvin Johnson
8. Yu Zhang

Citations in the corpus

• mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “Past work has also leveraged parallel data to improve multilingual text representations, e.g. with TLM (Conneau & Lample, 2019), explicit alignment (Hu et al., 2021) or nmT5 (Kale et al., 2021).”
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “Several pre-training tasks utilize the token-level alignments in parallel data to improve cross-lingual language models (Cao et al., 2020; Zhao et al., 2021; Hu et al., 2020a; Chi et al., 2021c).”
• Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, methodology): “Recently, pretrained LMs (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020) have shown impressive results and have proven to be useful in cross-lingual transfer settings (Libovickỳ et al., 2019; Hu et al., 2020b).”
• mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “Following Hu et al. (2020b), we compute the transfer gap scores over the sentence classification and question answering tasks.”