Polyglot Contextual Representations Improve Crosslingual Transfer

By Phoebe Mulcaire, Jungo Kasai, Noah A. Smith. 2019

73 citations recorded by the source corpus.

We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages. Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and non-contextual variants. Our results provide further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Subsequently, these contextual methods have been extended to produce multilingual representations by training a single model on text from multiple languages which have proven to be very effective for cross lingual transfer (Wu and Dredze, 2019; Mulcaire et al., 2019; Pires et al., 2019).”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “Prior work has shown that English is not the ideal pivot language for many under-resourced languages (Mulcaire et al., 2019; Conneau and Lample, 2019).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, positive): “For mBERT, we hypothesize a similar generalization across the 13 layers, as well as an abstraction away from a specific language with higher layers.”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, background): “Previous work has demonstrated that pre-trained multilingual models significantly help cross-lingual transfer on a wide range of NLP tasks such as cross-lingual named entity recognition (Pires et al., 2019; Mulcaire et al., 2019), zero-shot crosslingual dependency parsing (Schuster et al.”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “Other work has shown that mBERT outperforms previous state-of-the-art of cross-lingual transfer based on word embeddings on token-level NLP tasks (Wu and Dredze, 2019), and that adding character-level information (Mulcaire et al., 2019) and using multi-task learning (Huang et al.”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…training paradigm (Arivazhagan et al., 2019; Wang et al., 2020b), and how to further improve it by utilizing post-hoc alignment (Wang et al., 2020c; Cao et al., 2020), data balancing (Jean et al., 2019; Wang et al., 2020a), or calibrated training signal (Mulcaire et al., 2019; Huang et al., 2019a).”
Cross-Lingual Transfer Learning for Question Answering (2019, background): “It is also possible to train language model to obtain cross-lingual text representations and further improve QA performance in different languages using parallel data (Lample and Conneau, 2019; Mulcaire et al., 2019) or even without parallel data (Devlin et al., 2018) 1 .”
One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (2021, methodology): “…using pre-trained multilingual models such as mBERT or mT5 in many NLP tasks (e.g., entity linking: Botha et al., 2020; De Cao et al., 2021; semantic role labeling: Mulcaire et al., 2019b; Lyu et al., 2019; Fei et al., 2020; syntactic parsing: Mulcaire et al., 2019a; Kondratyuk and Straka, 2019).”