CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

By Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, Slav Petrov. 2018

433 citations recorded by the source corpus.

Every year, the Conference on Computational Natural Language Learning (CoNLL) features a shared task, in which participants train and test their learning systems on the same data sets. In 2018, one of two tasks was devoted to learning dependency parsers for a large number of languages, in a real-world setting without any gold-standard annotation on test input. All test sets followed a unified annotation scheme, namely that of Universal Dependencies. This shared task constitutes a 2nd edition—the first one took place in 2017 (Zeman et al., 2017); the main metric from 2017 has been kept, allowing for easy comparison, also in 2018, and two new main metrics have been used. New datasets added to the Universal Dependencies collection between mid-2017 and the spring of 2018 have contributed to increased difficulty of the task this year. In this overview paper, we define the task and the updated evaluation methodology, describe data preparation, report and analyze the main results, and provide a brief categorization of the different approaches of the participating systems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, neutral): “3 (Nivre et al., 2018; Zeman et al., 2018).”
FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP (2019, background): “, IMDB (Maas et al., 2011), TREC-6 (Voorhees and Harman, 2000), TREC-50 (Li and Roth, 2002), Universal De-pendencies (Zeman et al., 2018), WikiNER (Nothman et al., 2012) and WNUT-17 (Derczynski et al., 2017). treebank for English, simply execute these lines: # define dataset task = NLPTask.”
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages (2020, methodology): “For the character-level language models in the NER component, we pretrained them on a mix of the Common Crawl and Wikipedia dumps, and the news corpora released by the WMT19 Shared Task (Barrault et al., 2019), except for English and Chinese, for which we pretrained on the Google One Billion Word (C”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “Four monolingual fully trained ELMo models have been distributed for Japanese, Portuguese, German and Basque 5 ; 44 monolingual ELMo models 6 where also released by the HIT-SCIR team (Che et al., 2018) during the CoNLL 2018 Shared Task (Zeman et al., 2018), but their training sets where capped at 20”
We Need to Talk about Standard Splits (2019, methodology): “However, random split experiments could, for instance, be performed for the sub-tasks of the CoNLL-2018 shared task on multilingual parsing (Zeman et al., 2018).”
Polyglot Contextual Representations Improve Crosslingual Transfer (2019, background): “dency parsing (Zeman et al., 2018).”
Finding Universal Grammatical Relations in Multilingual BERT (2020, background): “Such languages have been shown to have better performance on recent shared task results on multilingual parsing (e.g. Zeman et al., 2018).”
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020, background): “The Universal Dependencies (UD) project 16 has released two different Indonesian corpora of relatively small size: (1) 5,593 sentences of UD-Indo-GSD (McDonald et al., 2013); 17 and (2) 1,000 sentences of UD-Indo-PUD (Zeman et al., 2018).”
Edinburgh Research Explorer A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages (neutral): “The organizers of the CoNLL 2018 UD shared task (Zeman et al., 2018) report that, in general, results on the task’s nine low-resource treebanks “are extremely low and the outputs are hardly useful for downstream applications.””
Multilingual is not enough: BERT for Finnish (2019, methodology): “A broad range of methods were applied to tagging these resources as a subtask in the recent CoNLL shared tasks in 2017 and 2018 (Zeman et al., 2018a), and we use the CoNLL 2018 versions (UD version 2.2) of these corpora to assure comparability with their results.”
From POS tagging to dependency parsing for biomedical event extraction (2018, methodology): “based character-level word embeddings, obtaining the best result on the CoNLL 2017 shared task on multilingual parsing (Zeman et al., 2017).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “This task has gained traction in large part because it has been a CoNLL shared task for the past two years [126].”
75 Languages, 1 Model: Parsing Universal Dependencies Universally (2019, methodology): “We propose UDify, a semi-supervised multi-task model for automatically producing UD annotations based on UDPipe Future, a winner of the CoNLL 2018 Shared Task on Multilingual Parsing from Raw Text to Universal Dependencies (Straka, 2018; Zeman et al., 2018).”
Construction and educational application of a linguistically grounded dependency treebank for Uyghur (2025, unknown): “We evaluated agreement using Unlabeled Attachment Score, Labeled Attachment Score, and Cohen’s Kappa coefficient (Zeman et al., 2018).”