Word Representation Models for Morphologically Rich Languages in Neural Machine Translation

By Ekaterina Vylomova, Trevor Cohn, Xuanli He, Gholamreza Haffari. 2016

52 citations recorded by the source corpus.

Out-of-vocabulary words present a great challenge for Machine Translation. Recently various character-level compositional models were proposed to address this issue. In current research we incorporate two most popular neural architectures, namely LSTM and CNN, into hard- and soft-attentional models of translation for character-level representation of the source. We propose semantic and morphological intrinsic evaluation of encoder-level representations. Our analysis of the learned representations reveals that character-based LSTM seems to be better at capturing morphological aspects compared to character-based CNN. We also show that hard-attentional model provides better character-level representations compared to vanilla one.

Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Machine Translation

Citations in the corpus

What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “Vylomova et al. (2016) also analyze different representations for morphologically-rich languages in MT, but do not directly measure the quality of the learned representations.”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, background): “Several studies investigated the ability of such models to learn linguistic properties at morphological (Vylomova et al., 2016; Belinkov et al., 2017a; Dalvi et al., 2017), syntactic (Shi et al.”
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks (2017, background): “Recent work has started exploring what kind of linguistic information such models learn on morphological (Vylomova et al., 2016; Belinkov et al., 2017; Dalvi et al., 2017) and syntactic levels (Shi et al.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “o Since neural networks generate representations in vector space, a common approach is to find their nearest neighbors and observe them for qualitative trends, for example to analyze morphological and semantic similarity of words [177, 326, 330].”
From Characters to Words to in Between: Do We Capture Morphology? (2017, background): “…can be exploited by composing the representations of characters (Ling et al., 2015; Kim et al., 2016), character n-grams (Sperr et al., 2013; Wieting et al., 2016; Bojanowski et al., 2016; Botha and Blun-som, 2014), bytes (Plank et al., 2016; Gillick et al., 2016), or combinations thereof…”
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, background): “, 2017a), as applied to morphological (Vylomova et al., 2017; Dalvi et al., 2017), semantic (Qian et al.”
On the Linguistic Representational Power of Neural Machine Translation Models (2019, background): “The language properties mainly analyzed are morphological (Qian, Qiu, and Huang 2016b; Vylomova et al. 2016; Belinkov et al. 2017a; Dalvi et al. 2017), semantic (Qian, Qiu, and Huang 2016b; Belinkov et al.”
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models (2018, background): “The language properties mainly analyzed are morphological (Qian, Qiu, and Huang 2016b; Vylomova et al. 2016), semantic (Qian, Qiu, and Huang 2016b) and syntactic (Shi, Padhi, and Knight 2016; Linzen, Dupoux, and Goldberg 2016; Conneau et al. 2018).”