XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

By Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei. 2021

139 citations recorded by the source corpus.

In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “Pre-training with Transformer [45] backbone networks has substantially advanced the state of the art across natural language processing [34, 10, 28, 22, 11, 36, 1, 7, 8, 4–6, The approaches of vision-language pre-training can be divided into two categories.”
DeepNet: Scaling Transformers to 1,000 Layers (2022, background): “2) For each layer, scale the weights of feed-forward net- worksaswellasthevalueprojectionandtheoutputpro-jectionofattentionlayersby √ log 2 N (or √ log 2 M ).”
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “They not only improve the performance on downstream multilingual NLP tasks (Conneau and Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Chi et al., 2021c), but also show an impressive cross-lingual transferability (Wu and Dredze, 2019; K et al.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “Encoder-Decoder or Encoder-only ELMo [80], BERT [28], RoBERTa [65], DistilBERT [90], BioBERT [57], XLM [54], Xlnet [119], ALBERT [55], ELECTRA [24], T5 [84], XLM-E [20], ST-MoE [133], AlexaTM [95] Training: Masked Language Models Model type: Discriminative (BERT-style) Pretrain task: Predict masked ”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Models like ELECTRA [5] and XLM-E [89] are pretrained using adversarial SSL. ELECTRA is pretrained using replaced token detection (RTD) while XLM-E is pretrained using multi-lingual replaced token detection(MRTD) and translation replaced token detection (TRTD).”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “Chi et al. (2021a) show that the information content between a A i and b B i can be maximized by minimizing the following InfoNCE (van den Oord et al., 2019) based loss function: where f ( a ) is the encoding of the sentence a as computed by the transformer.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “WavLM (Chen et al. 2021) is also built upon HuBERT with two additional techniques, namely gated relative position bias (Chi et al. 2021) and utterance mixing.”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “The Transformer is equipped with a convolution-based relative position embedding layer at the bottom, and the gated relative position bias [Chi et al., 2022] for better position information encoding.”
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing (2021, unknown): “The effectiveness of RTD is also verified by follow-up works, including CoCo-LM [10], XLM-E [11], and SmallBenchNLP [12].”
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators (2022, background): “This leads to significantly improved sample efficiency and effectiveness of ELECTRA-style pretrained models (Clark et al., 2020; Chi et al., 2021; Meng et al., 2021).”