Open-Source Tools for Morphology, Lemmatization, POS Tagging and Named Entity Recognition

By Jana Straková, Milan Straka, Jan Hajic. 2014

185 citations recorded by the source corpus.

We present two recently released opensource taggers: NameTag is a free software for named entity recognition (NER) which achieves state-of-the-art performance on Czech; MorphoDiTa (Morphological Dictionary and Tagger) performs morphological analysis (with lemmatization), morphological generation, tagging and tokenization with state-of-the-art results for Czech and a throughput around 10-200K words per second. The taggers can be trained for any language for which annotated data exist, but they are specifically designed to be efficient for inflective languages, Both tools are free software under LGPL license and are distributed along with trained linguistic models which are free for non-commercial use under the CC BY-NC-SA license. The releases include standalone tools, C++ libraries with Java, Python and Perl bindings and web services.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, methodology): “Here we opted for a feminine form whenever the system addresses itself, and formal address (mostly homonymous with plural) when addressing the user. the morphological generator of Strakov´a et al. (2014) to inﬂect the surface forms automatically and manually checked for errors.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “We processed the datasets using the MorphoDiTa part-of-speech tagger (Straková et al., 2014) to identify tokens, words (as opposed to punctuation tokens) and sentence boundaries.”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “For Czech, we pre-processed the data using MorphoDiTa (Straková et al., 2014) from http://ufal.”
New Language Pairs in TectoMT (2015, methodology): “The ﬁrst part, the surface syntactic analysis, consists of a rule-based sentence segmenter and tokenizer, followed by a part-of-speech tagger – we use MorphoDiTa (Straková et al., 2014) in the current version – and a dependency parser (McDonald et al., 2005; Novák and Žabokrtský, 2007).”
Evaluating the morphological competence of Machine Translation Systems (2017, methodology): “For Czech, we used the Morphodita analyzer (Straková et al., 2014).”
Alana v2: Entertaining and Informative Open-domain Social Dialogue using Ontologies and Entity Linking (2018, methodology): “POS tagger: we rely on the MorphoDiTa Part-of-Speech tagger (Straková et al., 2014) to extract POS tags for the user utterance.”
The University of Illinois submission to the WMT 2015 Shared Translation Task (2015, methodology): “To this end, we use MorphoDiTa (Straková et al., 2014) and the Stanford CoreNLP toolkit (Manning et al., 2014) to POS tag the Czech and English sentences, respectively.”
Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe (2017, methodology): “A guesser generates several (UPOS, XPOS, FEATS) triplets for each word according to its last four characters, and an averaged perceptron tagger with a fixed set of features disambiguates the generated tags (Straka et al., 2016; Straková et al., 2014).”
Gender Coreference and Bias Evaluation at WMT 2020 (2020, unknown): “For translated professions in Czech that were not resolved by the predeﬁned list, we use the automatic morphology tagger Mor-phoDiTa (Straková et al., 2014).”
Edinburgh’s Statistical Machine Translation Systems for WMT16 (2016, methodology): “For every Czech word in the training data, we obtained a set of morphological analyses using MorphoDiTa (Straková et al., 2014). From these analyses, we constructed a lexicon of feature structures. For constraint extraction, we used handwritten rules along the lines of those described in Williams (2”
SMT and Hybrid systems of the QTLeap project in the WMT16 IT-task (2016, methodology): “The TectoMT analysis pipeline is based on the annotation pipeline of the CzEng 1.0 corpus (Bo-jar et al., 2012) starting with a rule-based tokenizer and a statistical part-of-speech tagger (Strakov´a et al., 2014) and dependency parser (McDonald et al., 2005; Nov´ak and ˇZabokrtsk´y, 2007).”
LIUM Machine Translation Systems for WMT17 News Translation Task (2017, methodology): “PoS tagging is performed with Morphodita toolkit (Straková et al., 2014) as well as the reinﬂection to go from factored representation to word.”