From Characters to Words to in Between: Do We Capture Morphology?

By Clara Vania, Adam Lopez. 2017

103 citations recorded by the source corpus.

Words can be represented by composing the representations of subword units such as word segments, characters, and/or character n-grams. While such representations are effective and may capture the morphological regularities of words, they have not been systematically compared, and it is not understood how they interact with different morphological typologies. On a language modeling task, we present experiments that systematically vary (1) the basic unit of representation, (2) the composition of these representations, and (3) the morphological typology of the language modeled. Our results extend previous findings that character representations are effective across typologies, and we find that a previously unstudied combination of character trigram representations composed with bi-LSTMs outperforms most others. But we also find room for improvement: none of the character-level models match the predictive accuracy of a model with access to true morphological analyses, even when learned from an order of magnitude more data.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Indonesian: Indonesian is an Austronesian language characterized by reduplication of nouns, pronouns, adjectives, verbs, and numbers (Sneddon et al., 2012; Vania and Lopez, 2017), as well as prefixes, suffixes, infixes, and circumfixes.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “Vania and Lopez compare LMs (words, morphs, character n-grams, BPE) on several languages [83].”
Word Representation Models for Morphologically Rich Languages in Neural Machine Translation (2016, background): “With a few notable exceptions (Vania and Lopez, 2017; Heigold et al., 2017), there was no systematic investigation of the various modelling architectures.”
Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules? (2018, unknown): “Moreover, Vania and Lopez (2017) showed in the context of language modeling that training a BiLSTM on ground truth morphological features still yields better results than eight other character-based neural network architectures.”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Subword tokenization performs poorly on languages with reduplication (Vania and Lopez, 2017), while byte pair encoding does not align well with morphology (Bostrom and Durrett, 2020).”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “, 2018), phenomena such as code-switching and transliteration (Ball and Garrette, 2018), and rich morphology (Vania and Lopez, 2017), which had previously received special modeling including adaptor grammars (Botha and Blunsom, 2013).”
Modeling Vocabulary for Big Code Machine Learning (2019, background): “Vania and Lopez compare various subword decompositions (words, morphs, character n-grams, BPE) on several natural languages [69].”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “o Since neural networks generate representations in vector space, a common approach is to find their nearest neighbors and observe them for qualitative trends, for example to analyze morphological and semantic similarity of words [177, 326, 330].”
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, methodology): “Subword unit segmentations have been used to capture morphology [70] in a language modeling task but the network’s output were on the word level and thus unable output OOV entries.”