Alternating Language Modeling for Cross-Lingual Pre-Training

By Jian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, Ming Zhou. 2020

94 citations recorded by the source corpus.

Language model pre-training has achieved success in many natural language processing tasks. Existing methods for cross-lingual pre-training adopt Translation Language Model to predict masked words with the concatenation of the source sentence and its target equivalent. In this work, we introduce a novel cross-lingual pre-training method, called Alternating Language Modeling (ALM). It code-switches sentences of different languages rather than simple concatenation, hoping to capture the rich cross-lingual context of words and phrases. More specifically, we randomly substitute source phrases with target translations to create code-switched sentences. Then, we use these code-switched data to train ALM model to learn to predict words of different languages. We evaluate our pre-training ALM on the downstream tasks of machine translation and cross-lingual classification. Experiments show that ALM can outperform the previous pre-training methods on three benchmarks.1

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Zhiyuan Liu
3. Jianfeng Gao
4. Minlie Huang
5. Xipeng Qiu
6. Graham Neubig
7. Li Dong
8. Mohit Bansal

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “REALM (Guu et al., 2020) is a pioneer to explore how to construct a sustainable external memory for Transformers.”
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, background): “ALM (Yang et al., 2020) extends TLM to code-switched sequences obtained from translation pairs.”
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “ALM (Yang et al., 2020) utilizes translation pairs to construct code-switched sequences as input.”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “mT6 [91], XLM [63], infoXLM [90], Unicoder [103], ALM [104], XLM-E [89]”
• VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation (2020, methodology): “Unicoder (Huang et al., 2019), ALM (Yang et al., 2020), and InfoXLM (Chi et al., 2020b) propose new pre-training tasks to better utilize the bilingual data.”
• mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “ALM (Yang et al., 2020) converts the translation pairs into code-switched sequences as the training examples.”
• nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021, methodology): “Alternating language modeling (Yang et al., 2020) and back-translation masked language modeling (Ouyang et al., 2020) used code-switched sentences and back-translation respectively to utilize parallel data.”
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, background): “ALM (Yang et al., 2020) improves the model’s transferability by enabling the model to learn cross-lingual code-switch sentences.”
• A Primer on Pretrained Multilingual Language Models (2021, unknown): “To make the task challenging, they train a lightweight paraphrase detection model and sample Z that is very close to X but is not equal to Y Alternating Language Model (ALM).”