Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning

By Zihan Liu, Genta Indra Winata, Andrea Madotto, Pascale Fung. 2020

20 citations recorded by the source corpus.

Recently, fine-tuning pre-trained cross-lingual models (e.g., multilingual BERT) to downstream cross-lingual tasks has shown promising results. However, the fine-tuning process inevitably changes the parameters of the pre-trained model and weakens its cross-lingual ability, which could lead to sub-optimal performances. To alleviate this issue, we leverage the idea of continual learning to preserve the original cross-lingual ability of the pre-trained model when we fine-tune it to downstream cross-lingual tasks. The experiment on the cross-lingual sentence retrieval task shows that our fine-tuning approach can better preserve the cross-lingual ability of the pre-trained model. In addition, our method achieves better performance than other fine-tuning baselines on zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bing Liu
2. Zixuan Ke
3. Hu Xu
4. Lei Shu
5. Bin Liu
6. Colin Raffel
7. Noah Constant
8. Mohit Bansal

In Natural Language Processing

1. Hu Xu
2. Lei Shu
3. Bing Liu
4. Zixuan Ke
5. Noah Constant
6. Colin Raffel
7. Rami Al-Rfou
8. Mihir Kale

Citations in the corpus

• mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Popular models of this type are mBERT (Devlin, 2018), mBART (Liu et al., 2020), and XLM-R (Conneau et al., 2019), which are multilingual variants of BERT (Devlin et al., 2018), BART (Lewis et al., 2019a), and RoBERTa (Liu et al., 2019), respectively.”
• Continual Training of Language Models for Few-Shot Learning (2022, background): “…2019), language learning (Li et al., 2019), sentence embedding (Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classification (Ke et al., 2021a,b; Sun et al., 2020; Huang et al., 2021; Chuang…”
• Memory Efficient Continual Learning with Transformers (2022, background): “Recent work studied catastrophic forgetting [55, 10, 27, 35] and incremental learning [66] for NLP and CV [32, 15].”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “For example, CL models have been proposed for sentiment analysis [23, 24, 37, 43], dialogue slot filling [53], language modeling [58, 7], language learning [31], sentence embedding [33], machine translation [25], cross-lingual modeling [35], and question answering [12].”
• A Primer on Pretrained Multilingual Language Models (2021, unknown): “…Sabet et al., 2020; Chen et al., 2020; Zenkel et al., 2020; Dou and Neubig, 2021; Imamura and Sumita, 2019; Ma et al., 2020; Zhu et al., 2020; Liu et al., 2020b; Xue et al., 2021) which use MLLMs for downstream bilingual tasks such as unsupervised machine translation, cross-lingual word…”
• Memory Efficient Continual Learning for Neural Text Classification (2022, background): “Recent work studied catastrophic forgetting (Sun et al., 2019; Chuang et al., 2020; Ke et al., 2020; Liu et al., 2020) and incremental learning (Xia et al., 2021) for NLP. Pa-sunuru et al. (2021) focus on the few-shot setting where only a few data points are available for each task.”