InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

By Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, M. Zhou. 2020

442 citations recorded by the source corpus.

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Note that unlike our model, InfoXLM (Chi et al., 2020) beneﬁts from parallel training data, while X-STILTs (Phang et al., 2020) leverages labeled data from tasks similar to the target task.”
Knowledge Neurons in Pretrained Transformers (2021, background): “Besides, an interesting direction is to figure out how knowledge neurons work in multilingual pretrained Transformers (Conneau and Lample, 2019; Conneau et al., 2020; Chi et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “In addition, XNLG optimizes parameters in a two-stage manner.”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Contrastive learning has recently attracted a surge of interest and shown promising results in various areas including computer vision [18, 6], natural language processing [11, 15], audio processing [49, 42], graph [20, 44] and multimodal data [4, 51].”
Better Fine-Tuning by Reducing Representational Collapse (2020, methodology): “The current state of the art required a novel pre-training method to reach the same numbers as (Chi et al., 2020).”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “We believe the cross-lingual contrast (Chi et al., 2021b) task explicitly learns the sentence representations, which makes I NFO XLM more effective for the cross-lingual sentence retrieval task.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The benefits of sentence level pretraining task are usually observed on some specific tasks (Chi et al., 2020; Lewis et al., 2020) such as modeling long-form texts (Ravula et al.”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, unknown): “Besides monolingual pretrained models, multilingual pretrained models (Devlin et al., 2018; Lample and Conneau, 2019; Chi et al., 2019; Conneau et al., 2019; Chi et al., 2020) also advance the state-of-the-art on cross-lingual understanding and generation.”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, methodology): “InfoXLM (Chi et al., 2020b) proposes a novel cross-lingual contrast to enhance the cross-lingual transferability of the pretrained encoders.”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “Our primary submission is an ensemble between a C OMET Estimator model trained on top of XLM-R using DA from 2017 to 2020 and a sequence tagging model, such as the one described above, trained on top of InfoXLM (Chi et al., 2021).”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “Chi et al. (2021a) show that the information content between a A i and b B i can be maximized by minimizing the following InfoNCE (van den Oord et al., 2019) based loss function: where f ( a ) is the encoding of the sentence a as computed by the transformer.”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “T-URLv2 and HiCTL (Wei et al., 2021), based on InfoXLM (Chi et al., 2021) and XLM-R respectively, employ contrastive losses.”