Compositional Morphology for Word Representations and Language Modelling

By Jan A. Botha, Phil Blunsom. 2014

267 citations recorded by the source corpus.

This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model. Our approach is evaluated in the context of log-bilinear language models, rendered suitably efficient for implementation inside a machine translation decoder by factoring the vocabulary. We perform both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that our model learns morphological representations that both perform well on word similarity tasks and lead to substantial reductions in perplexity. When used for translation into morphologically rich languages with large vocabularies, our models obtain improvements of up to 1.2 BLEU points relative to a baseline system using back-off n-gram models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Neural Machine Translation of Rare Words with Subword Units (2015, background): “Various techniques have been proposed to produce ﬁxed-length continuous word vectors based on characters or morphemes (Luong et al., 2013; Botha and Blunsom, 2014; Ling et al., 2015a; Kim et al., 2015).”
Character-Aware Neural Language Models (2015, methodology): “Unlike previous works that utilize subword information via morphemes (Botha and Blunsom 2014; Lu-ong, Socher, and Manning 2013), our model does not require morphological tagging as a pre-processing step.”
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation (2015, methodology): “For instance, the work in (Botha and Blunsom, 2014) shows that using morphological analyzers to generate morphological features, such as stems, prefixes and suffixes can be used to learn better representations for words.”
Character-based Neural Machine Translation (2016, background): “Improvements up to 3 BLEU points are obtained in the German-English WMT task.”
What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation (2015, background): “They have been shown to be beneficial for several tasks such as machine translation (Botha and Blunsom, 2014), parsing (Lei et al.”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “Botha and Blunsom (2014) suggest to model the embedding vector of a word as a sum of the word-specific vector if such vector is available, with vectors for the different morphological components that comprise it (the components are derived using Morfessor (Creutz & Lagus, 2007), an unsupervised morp”
Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs (2015, background): “Botha and Blunsom (2014) show that stems, preﬁxes and sufﬁxes can be used to learn useful word representations but relying on an external morphological analyzer.”
Suggesting accurate method and class names (2015, methodology): “Related to our subtoken model is the work of Botha and Blunsom [12] that integrate compositional morphological representations of words into a log-bilinear LM but the morphological features are only used in the context of an LM.”
Byte-Level Machine Reading Across Morphologically Varied Languages (2018, methodology): “To overcome the issues caused by a lack of vocabulary coverage and sparse training data as outlined above, models have been proposed that work at the sub-word level, e.g., morphemes (Luong, Socher, and Manning 2013; Botha and Blunsom 2014).”
Morphological Word-Embeddings (2019, methodology): “Finally, most similar to our model, Botha and Blunsom (2014) introduced the additive logbilinear model (LBL++).”
Effective Subword Segmentation for Text Comprehension (2018, background): “informed word representations [2], [4], [7], [18], aiming at better capturing syntactic, lexical and morphological information.”
Finding Better Subword Segmentation for Neural Machine Translation (2018, background): “Some other work segmented words based on morphological information to obtain the representation of OOV words [2,5], which can be called linguistically motivated method.”
From Characters to Words to in Between: Do We Capture Morphology? (2017, methodology): “, 2016), character n-grams (Sperr et al., 2013; Wieting et al., 2016; Bojanowski et al., 2016; Botha and Blunsom, 2014), bytes (Plank et al.”
Pragmatic Neural Language Modelling in Machine Translation (2014, background): “It has been shown that neural language models can improve translation quality when used as additional features in a decoder (Vaswani et al., 2013; Botha and Blunsom, 2014; Baltescu et al., 2014; Auli and Gao, 2014) or if used for n-best list rescoring (Schwenk, 2010; Auli et al., 2013).”