75 Languages, 1 Model: Parsing Universal Dependencies Universally

By D. Kondratyuk. Conference on Empirical Methods in Natural Language Processing, 2019

294 citations recorded by the source corpus.

We present UDify, a multilingual multi-task model capable of accurately predicting universal part-of-speech, morphological features, lemmas, and dependency trees simultaneously for all 124 Universal Dependencies treebanks across 75 languages. By leveraging a multilingual BERT self-attention model pretrained on 104 languages, we found that fine-tuning it on all datasets concatenated together with simple softmax classifiers for each UD task can meet or exceed state-of-the-art UPOS, UFeats, Lemmas, (and especially) UAS, and LAS scores, without requiring any recurrent or language-specific components. We evaluate UDify for multilingual learning, showing that low-resource languages benefit the most from cross-linguistic annotations. We also evaluate for zero-shot learning, with results suggesting that multilingual training provides strong UD predictions even for languages that neither UDify nor BERT have ever been trained on.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/31c872514c28a172f7f0221c8596aa5bfcdb9e98

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The Unstoppable Rise of Computational Linguistics in Deep Learning (2020, background): “cent work on dependency parsing (Kondratyuk and Straka, 2019; Mohammadshahi and Henderson, 2019, 2020).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, 2021a), Dependency Parsing (Kondratyuk and Straka, 2019), Question Answering (Huang et al.”
UDapter: Language Adaptation for Truly Universal Dependency Parsing (2020, methodology): “We train and test our model multilingually on the concatenation of 13 syntactically diverse high-resource languages that were used by Kulmizev et al. (2019), and also evaluate on 30 genuinely low-resource languages.”
Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (2020, methodology): “Most recent DP work (Kondratyuk and Straka, 2019; Üstün et al., 2020) replaces the Bi-LSTM encoder with multilingual BERT’s transformer, reporting state-of-the-art parsing performance.”
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, methodology): “We experimented with commonly used multilingual parsers like UDPipe (Straka and Straková, 2017), UDify (Kondratyuk and Straka, 2019), and Stanza (Qi et al., 2020), settling on Stanza for its superior performance in preliminary experiments.”
Multilingual is not enough: BERT for Finnish (2019, methodology): “…of natural language processing (NLP) tasks ranging from sequence labeling tasks such as part-of-speech (POS) tagging and named entity recognition (NER) (Peters et al., 2018b) to dependency parsing (Kondratyuk and Straka, 2019) and natural language understanding (NLU) tasks (Devlin et al., 2018).”
One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (2021, methodology): “…using pre-trained multilingual models such as mBERT or mT5 in many NLP tasks (e.g., entity linking: Botha et al., 2020; De Cao et al., 2021; semantic role labeling: Mulcaire et al., 2019b; Lyu et al., 2019; Fei et al., 2020; syntactic parsing: Mulcaire et al., 2019a; Kondratyuk and Straka, 2019).”
X -stance: A Multilingual Multi-Target Dataset for Stance Detection (2020, unknown): “We choose a multi-source baseline because M-B ERT is known to ben-eﬁt from multilingual training data both in a supervised and in a cross-lingual scenario (Kondratyuk and Straka, 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “• Taking more layers into account: learning a complementary representation of the information in deep and output layers (Yang and Zhao, 2019), using a weighted combination of all layers instead of the final one (Su and Cheng, 2019; Kondratyuk and Straka, 2019), and layer dropout (Kondratyuk and Stra”
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020, methodology): “5,6 We add the following task-speciﬁc architectures on top of MMTs: for DEP we add the biafﬁne parsing head (Dozat and Manning, 2017; Kondratyuk and Straka, 2019); for POS, we attach a simple feed-forward token-level classiﬁer; for NER, we feed MMT’s token-level outputs to a CRF classi-ﬁer, similar…”
CamemBERT: a Tasty French Language Model (2019, methodology): “It is relevant to compare CamemBERT to UDify on those tasks because UDify is the work that pushed the furthest the performance in fine-tuning end-to-end a BERT-based model on downstream POS tagging and dependency parsing.”
COMET: A Neural Framework for MT Evaluation (2020, methodology): “In order to avoid overfitting to the information contained in any single layer, we used layer dropout (Kondratyuk and Straka, 2019), in which with a probability p the weight α(i) is set to −∞.”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “We obtain the state of the art for the three metrics in each of the languages with the UDPipe Table 5: Scores from UDPipe 2.0 (from Kondratyuk and Straka, 2019), the previous state-of-the-art models UDPipe 2.0+mBERT (Straka et al., 2019) and UDify (Kondratyuk and Straka, 2019), and our…”
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020, methodology): “The universal fine-tuning approach (Kondratyuk and Straka, 2019) yields similar performance as BiAffine + fastText, while augmenting BiAffine with MBERT and MALAYBERT yields lower UAS and LAS scores than INDOBERT.”
Small and Practical BERT Models for Sequence Labeling (2019, unknown): “Kondratyuk (2019) shows similar results.”
Coreferential Reasoning Learning for Language Representation (2020, background): “, 2020a); and (3) exploring multilingual learning (Conneau and Lample, 2019; Tan and Bansal, 2019; Kondratyuk and Straka, 2019) or multimodal learning (Lu et al.”