Massively Multilingual Transfer for NER

By Afshin Rahimi, Yuan Li, Trevor Cohn. 2019

237 citations recorded by the source corpus.

In cross-lingual transfer, NLP models over one or more source languages are applied to a low-resource target language. While most prior work has used a single source model or a few carefully selected models, here we consider a “massive” setting with many such models. This setting raises the problem of poor transfer, particularly from distant languages. We propose two techniques for modulating the transfer, suitable for zero-shot or few-shot learning, respectively. Evaluating on named entity recognition, we show that our techniques are much more effective than strong baselines, including standard ensembling, and our unsupervised method rivals oracle selection of the single best individual model.

Subfields: Natural Language Processing, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Information Extraction

Citations in the corpus

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “…approaches have been evaluated on a wide range of tasks, including dependency parsing (Schuster et al., 2019), named entity recognition (Rahimi et al., 2019), sentiment analysis (Barnes et al., 2018), natu-ral language inference (Conneau et al., 2018b), document classiﬁcation (Schwenk…”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “We compare our framework against state-of-the-art cross-lingual transfer methods on the standard WikiANN NER dataset (Pan et al., 2017; Rahimi et al., 2019), relying on a representative set of ty-pologically diverse languages which includes high-resource, low-resource, as well as languages un-seen…”
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, background): “5 (Zeman et al., 2019), NER named entity recognition on the Wikiann (Pan et al., 2017; Rahimi et al., 2019) dataset, cross-lingual natu-ral language inference on XNLI (Conneau et al., 2018), cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019a), and cross-lingual…”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, methodology): “For NER, we use the WikiAnn (Pan et al., 2017) dataset, partitioned into train, development, and test portions by Rahimi et al. (2019).”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, background): “Cross-lingual transfer has proven effective in many NLP tasks including dependency parsing (Schuster et al., 2019), named entity recognition (Rahimi et al., 2019), sentiment analysis (Barnes et al., 2019), document classification (Schwenk and Li, 2018), and question…”
On the Representation Collapse of Sparse Mixture of Experts (2022, background): “5 part-of-speech tagging (Zeman et al., 2019), WikiAnn named entity recognition (Pan et al., 2017; Rahimi et al., 2019), natural language inference (XNLI; Conneau et al. 2018), paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019), and question answering on MLQA (Lewis et al.,…”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Instead of having a set of tasks, Rahimi et al. (2019) built an ensemble of language-specific NER models which are then weighted depending on the zero- or few-shot target language.”
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, background): “…sequence labeling mainly focused on cross-lingual transfer learning to improve the performance of low-resource or zero-resource languages (Johnson et al., 2019; Huang et al., 2019a; Rahimi et al., 2019; Huang et al., 2019b; Keung et al., 2019), but their work still trains monolingual models.”
Intriguing Properties of Compression on Multilingual Models (2022, methodology): “We focus our experiments on the 40 languages from the XTREME benchmark (Hu et al., 2020), with train-test splits deﬁned by Rahimi et al. (2019).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “We rely on the following NER datasets: CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003), FiNER (Ruokolainen et al., 2020), Chinese Literature (Xu et al., 2017), KMOU NER29, WikiAnn (Pan et al., 2017; Rahimi et al., 2019).”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “We use the balanced train, dev, and test splits from Rahimi et al. (2019).”
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020, methodology): “We also observe the gradual unfreezing scheme to improve (Devlin et al., 2019) 91.86 2.7 MMNER (Rahimi et al., 2019) 89.20 2.8 XtremeDistil (ours) 88.64 3.8 Table 5: F 1 -score comparison of different models with standard deviation across 41 languages. both stage-wise distillation strategies signiﬁc”
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “, 2020), we consider six downstream tasks included by XTREME: the named entity recognition (NER) task on the WikiAnn (Pan et al., 2017; Rahimi et al., 2019) dataset in 40 languages, question answering (QA) task on MLQA (Lewis et al.”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, methodology): “Following Hu et al. (2020), we use the version with balanced train, development, and test splits from Rahimi et al. (2019).”
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020, methodology): “We resort to the NER WikiANN dataset from Rahimi et al. (2019).”