English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

By Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, Samuel R. Bowman. 2020

69 citations recorded by the source corpus.

Intermediate-task training—fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task—often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tasks and moderate improvements on question-answering target tasks. MNLI, SQuAD and HellaSwag achieve the best overall results as intermediate tasks, while multi-task intermediate offers small additional improvements. Using our best intermediate-task models for each target task, we obtain a 5.4 point improvement over XLM-R Large on the XTREME benchmark, setting the state of the art as of June 2020. We also investigate continuing multilingual MLM during intermediate-task training and using machine-translated intermediate-task data, but neither consistently outperforms simply performing English intermediate-task training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Note that unlike our model, InfoXLM (Chi et al., 2020) beneﬁts from parallel training data, while X-STILTs (Phang et al., 2020) leverages labeled data from tasks similar to the target task.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, neutral): “, 2018) (Stickland and Murray, 2019) (Phang et al., 2020)”
Merging Models with Fisher-Weighted Averaging (2021, background): “Recently, it has been shown that training on an “intermediate” task between pre-training and ﬁne-tuning can further boost performance through additional transfer of capabilities from the intermediate task [47, 60, 51, 48].”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “Fine-tuning on sentence-level tasks (Phang et al., 2020; Fang et al., 2021) can mitigate this.”
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, methodology): “…2019; Vu et al., 2020) and P ROMPT T UNING (Vu et al., 2022), and has been successfully applied to the zero-shot cross-lingual transfer setting (Phang et al., 2020; Maurya et al., 2021) vised task from the target language is helpful in conjunction with freezing some model components for M ODEL…”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “, 2018), and recently adopted this idea to cross-lingual learning, where English is used as intermediate-task for zero-shot transfer (Phang et al., 2020).”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “, 2020a) 24 16 1024 559M MLM CommonCrawl 7 7 100 250K X-STILTS (Phang et al., 2020) 24 16 1024 559M MLM CommonCrawl 7 X 100 250K HiCTL-base (Wei et al.”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “Relevant work also studies intermediate fine-tuning using crosslingual supervision (Phang et al., 2020; Moghe et al., 2021).”