Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

By Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, M. Zhou. 2019

242 citations recorded by the source corpus.

We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM , three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, background): “Multilingual Pre-training via RAS Recent work proves that cross-lingual language model pre-training could be a more effective way to representation learning (Conneau and Lample, 2019; Huang et al., 2019).”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “Inspired by BERT and some recent cross-lingual pre-trained models. such as XLM (Lample and Con-neau 2019) and Unicoder (Huang et al., 2019), a cross-modal pre-training framework is designed to model the relationships between visual and linguistic contents and learn their joint representations.”
Injecting Numerical Reasoning Skills into Language Models (2020, background): “Designing pre-training tasks to teach LMs additional skills has been applied by Huang et al. (2019), who designed cross-lingual pre-training tasks to teach better mappings between languages, and Lee et al. (2019), who introduced the Inverse Cloze Task to pre-train an information retriever.”
Unsupervised Cross-lingual Representation Learning at Scale (2019, background): “They uncover the high and low resource trade-off and the problem of capacity dilution (Johnson et al., 2017; Tan et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “…pre-training objective where words are masked in parallel sentence pairs (the model can attend to one or both sentences to solve the prediction task); • Huang et al. (2019) combine 5 pre-training tasks (monolingual and cross-lingual MLM, translation language modeling, cross-lingual word…”
On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “Extensions to mBERT include scaling it up and incorporating parallel data (Conneau and Lample, 2019), adding auxiliary pre-training tasks (Huang et al., 2019), and encouraging representations of translations to be similar (Cao et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, methodology): “Existing pipeline methods include GPipe (Huang et al., 2019b) which can send smaller parts of samples within a mini-batch to different nodes, and Tera Pipe (Li et al., 2021) which can apply token-level pipeline mechanisms for Transformer-based models to make each token in a sequence be processed by…”
Pre-trained models for natural language processing: A survey (2020, background): “Multilingual XLU mBERT [36], Unicoder [68], XLM [27], XLM-R [28], MultiFit [42]”
XGPT: Cross-modal Generative Pre-Training for Image Captioning (2020, positive): “Following Unicoder [21] , we calculate task-specific loss in turns and update the model for each task in every pre-training iteration.”
XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation (2020, background): “…are explored recently, including monolingual pre-trained models (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019b; Dong et al., 2019; Lewis et al., 2019a), multilingual and cross-lingual pre-trained models (Devlin et al., 2019; Conneau and Lample, 2019; Huang et…”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “…outperforms previous state-of-the-art of cross-lingual transfer based on word embeddings on token-level NLP tasks (Wu and Dredze, 2019), and that adding character-level information (Mulcaire et al., 2019) and using multi-task learning (Huang et al., 2019) can improve cross-lingual performance.”
Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning (2020, methodology): “, 2020), and utilizing more tasks in the pre-training stage (Huang et al., 2019).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “However, to our knowledge, none of existing work has conducted large-scale evaluations of ChatGPT for multiple and diverse languages/tasks as we do.”
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding (2020, background): “For example, mBERT (Devlin et al. 2019), Uni-coder (Huang et al. 2019) and XLM (Lample and Conneau 2019) have achieved strong performance on many cross-lingual tasks by successfully transferring knowledge from source language to a target one.”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, background): “There are also other work (Huang et al., 2019; Luo et al., 2020) to explore different pretraining tasks to further improve the performance of pretrained multilingual model.”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…training paradigm (Arivazhagan et al., 2019; Wang et al., 2020b), and how to further improve it by utilizing post-hoc alignment (Wang et al., 2020c; Cao et al., 2020), data balancing (Jean et al., 2019; Wang et al., 2020a), or calibrated training signal (Mulcaire et al., 2019; Huang et al., 2019a).”