On Learning Universal Representations Across Languages

By Xiangpeng Wei, Yue Hu, Rongxiang Weng, Luxi Xing, Heng Yu, Weihua Luo. 2020

93 citations recorded by the source corpus.

Recent studies have demonstrated the overwhelming advantage of cross-lingual pre-trained models (PTMs), such as multilingual BERT and XLM, on cross-lingual NLP tasks. However, existing approaches essentially capture the co-occurrence among tokens through involving the masked language model (MLM) objective with token-level cross entropy. In this work, we extend these approaches to learn sentence-level representations, and show the effectiveness on cross-lingual understanding and generation. We propose Hierarchical Contrastive Learning (HiCTL) to (1) learn universal representations for parallel sentences distributed in one or multiple languages and (2) distinguish the semantically-related words from a shared cross-lingual vocabulary for each sentence. We conduct evaluations on three benchmarks: language understanding tasks (QQP, QNLI, SST-2, MRPC, STS-B and MNLI) in the GLUE benchmark, cross-lingual natural language inference (XNLI) and machine translation. Experimental results show that the HiCTL obtains an absolute gain of 1.0%/2.2% accuracy on GLUE/XNLI as well as achieves substantial improvements of +1.7-+3.6 BLEU on both the high-resource and low-resource English-to-X translation tasks over strong baselines. We will release the source codes as soon as possible.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Zhe Gan
2. Mohit Bansal
3. Linjie Li
4. Graham Neubig
5. Jianfeng Gao
6. Li Dong
7. Sebastian Ruder
8. Xu Han

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “HICTL (Wei et al., 2021) extends the idea of using contrastive learning to learn both sentence-level and word-level cross-lingual representations.”
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Publicly accessible large-scale multi-task benchmarks [12, 62, 61, 23, 37] have facilitated recent advances [16, 72, 42, 71, 64, 18, 47] in NLP.”
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “Contrastive learning is also used in Hictl (Wei et al., 2021) and post-pretrained multilingual BERT (Pan et al.”
• A Primer on Pretrained Multilingual Language Models (2021, unknown): “Wei et al. (2021) also uses an InfoNCE based contrastive loss (CTL) and extends it to learn both sentence level and word level cross-lingual representations.”
• XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, background): “T-URLv2 and HiCTL (Wei et al., 2021), based on InfoXLM (Chi et al., 2021) and XLM-R respectively, employ contrastive losses.”
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, background): “H ICTL (Wei et al., 2020) learns cross-lingual semantic representation from multiple facets (at word-levels and sentence-levels) to improve the performance of cross-lingual models.”