UDPipe: Trainable Pipeline for Processing CoNLL-U Files Performing Tokenization, Morphological Analysis, POS Tagging and Parsing

By Milan Straka, Jan Hajic, Jana Straková. International Conference on Language Resources and Evaluation, 2016

472 citations recorded by the source corpus.

Automatic natural language processing of large texts often presents recurring challenges in multiple languages: even for most advanced tasks, the texts are first processed by basic processing steps – from tokenization to parsing. We present an extremely simple-to-use tool consisting of one binary and one model (per language), which performs these tasks for multiple languages without the need for any other external data. UDPipe, a pipeline processing CoNLL-U-formatted files, performs tokenization, morphological analysis, part-of-speech tagging, lemmatization and dependency parsing for nearly all treebanks of Universal Dependencies 1.2 (namely, the whole pipeline is currently available for 32 out of 37 treebanks). In addition, the pipeline is easily trainable with training data in CoNLL-U format (and in some cases also with additional raw corpora) and requires minimal linguistic knowledge on the users’ part. The training code is also released.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c5f5bb3131f5f1082bd82cce8a0ac08dea1e9366

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “Next, we follow Pyysalo et al. (2020) and utilize UDPipe (Straka et al., 2016) parsers pretrained on UD data13 to segment the extracted text passages into texts with document, sentence, and word boundaries.”
CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies (2018, background): “However, for teams who wanted to concentrate on one or more subtasks, segmentation and morphology predicted by the baseline UDPipe system (Straka et al., 2016) was made available just like last year.”
Are All Languages Equally Hard to Language-Model? (2018, methodology): “We use a long short-term memory (LSTM) LM (Sundermeyer et al., 2012), identical to that of Zaremba et al. (2014), but at the character-level.”
Twitter Universal Dependency Parsing for African-American and Mainstream American English (2018, background): “UDPipe (Straka et al., 2016) is a neural pipeline containing a tokenizer, morphological analyzer,”
Findings of the 2017 DiscoMT Shared Task on Cross-lingual Pronoun Prediction (2017, methodology): “For Spanish, we used UDPipe (Straka et al., 2016), which includes universal POS tags and a lemmatizer.”
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs (2020, methodology): “We use UDPipe to tokenize each corpus (Straka et al., 2016).”
Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe (2017, methodology): “0 (Straka et al., 2016)(4) is a trainable pipeline performing sentence segmentation, tokenization, POS tagging, lemmatization and dependency parsing.”
Polyglot Contextual Representations Improve Crosslingual Transfer (2019, methodology): “The Traditional Chinese corpus was already pre-segmented by UDPipe (Ginter et al., 2017; Straka et al., 2016).”
Cross-lingual and cross-domain discourse segmentation of entire documents (2017, methodology): “We use the pre-trained UDPipe models to postag the documents for all languages.”
A survey on sentiment analysis methods, applications, and challenges (2022, background): “For instance, “This mobile is amazing” may be tagged as follows: (Straka et al. 2016) This :determiner, mobile:noun, is:verb, amazing:adjective.”
CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies (2017, methodology): “Participants were free to use any part of the model in their systems – for all test sets, we provided UDPipe processed variants in addition to raw text inputs.”
Robust Multilingual Part-of-Speech Tagging via Adversarial Training (2017, methodology): “Dozat et al. (2017) empirically show that their dependency parser gains signiﬁcant improvements by using POS tags predicted by a Bi-LSTM POS tagger, while POS tags predicted by the UDPipe tagger (Straka et al., 2016) do not contribute to parsing performance as much.”
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color (2021, methodology): “To determine which factors could predict alignment score, we extract color term log frequency, part-of-speech tag (POS), dependency relation (DREL), and dependency tree head (HEAD) statistics for all color terms from a dependency-parsed (Straka et al., 2016) common crawl corpus.”