Choosing Transfer Languages for Cross-Lingual Learning
By Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma and 3 more. 2019
265 citations recorded by the source corpus.
Cross-lingual transfer, where a high-resource transfer language is used to improve the accuracy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low-resource languages. However, given a particular task language, it is not clear which language to transfer from, and the standard strategy is to select languages based on ad hoc criteria, usually the intuition of the experimenter. Since a large number of features contribute to the success of cross-lingual transfer (including phylogenetic similarity, typological properties, lexical overlap, or size of available data), even the most enlightened experimenter rarely considers all these factors for the particular task at hand. In this paper, we consider this task of automatically selecting optimal transfer languages as a ranking problem, and build models that consider the aforementioned features to perform this prediction. In experiments on representative NLP tasks, we demonstrate that our model predicts good transfer languages much better than ad hoc baselines considering single features in isolation, and glean insights on what features are most informative for each different NLP tasks, which may inform future ad hoc selection even without use of our method.
Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Sebastian Ruder
- Mohit Bansal
- Ivan Vulic
- Luke S. Zettlemoyer
- Samuel R. Bowman
- Noah A. Smith
- Yulia Tsvetkov
In Natural Language Processing
- Graham Neubig
- Sebastian Ruder
- Orhan Firat
- Yulia Tsvetkov
- Samuel R. Bowman
- Ivan Vulic
- Melvin Johnson
- Mohit Bansal
In Machine Translation
- Graham Neubig
- Yulia Tsvetkov
- Anoop Kunchukuttan
- Heike Adel
- Sebastian Ruder
- Melvin Johnson
- Anders Søgaard
- Ivan Vulic
Citations in the corpus
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “Although English is not generally the best source language for cross-lingual transfer for all target languages (Lin et al., 2019), this is still the most practically useful setting.”
- Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, background): “Similar work (Lin et al., 2019b) has been done on cross-lingual transfer—the analogous challenge of transferring learned knowledge from a high-resource to a low-resource language.”
- Balancing Training for Multilingual Neural Machine Translation (2020, background): “Second, this heuristic ignores factors other than data size that affect the interaction between different languages, despite the fact that language similarity has been empirically proven important in examinations of cross-lingual transfer learning (Wang and Neubig, 2019; Lin et al., 2019).”
- A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “gual model can be trained on the target task in a high-resource language and afterwards, applied to the unseen target languages, such as for named entity recognition (Lin et al., 2019; Hvingelby et al., 2020), reading comprehension (Hsu et al.”
- Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Other factors that inluence TL include the size of the parent and child corpora, domains of the parent and child data, the number of shared words (vocabulary overlap) and the language script [4, 38, 102, 103, 120].”
- Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “For morphosyntactic cross-lingual work, simple overlap metrics are indicative (Üstün et al., 2019; Lin et al., 2019).”
- Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “LSTMs are also sensitive to word order and perform worse on agreement prediction in Basque, which is both morphologically richer and has a relatively free word order (Ravfogel et al., 2018) compared to English (Linzen et al., 2016).”
- Towards multidomain and multilingual abusive language detection: a survey (2021, background): “Cross-lingual transfer learning is the common approach to transfer knowledge from one language (usually with more available resources) to another language (usually with less resources) [69, 126].”
- Should All Cross-Lingual Embeddings Speak English? (2019, unknown): “In the future, we will investigate a hub language ranking / selection model a la Lin et al. (2019).”
- A Primer on Pretrained Multilingual Language Models (2021, background): “To evaluate the effectiveness of this default choice and compare with other high resource alternatives, Lin et al. (2019) formulate the task of choosing the best source as a learning to rank problem.”