Alternative structures for character-level RNNs

By Piotr Bojanowski, Armand Joulin, Tomas Mikolov. 2015

53 citations recorded by the source corpus.

Recurrent neural networks are convenient and efficient models for language modeling. However, when applied on the level of characters instead of words, they suffer from several problems. In order to successfully model long-term dependencies, the hidden representation needs to be large. This in turn implies higher computational costs, which can become prohibitive in practice. We propose two alternative structural modifications to the classical RNN model. The first one consists on conditioning the character level representation on the previous word representation. The other one uses the character history to condition the output probability. We evaluate the performance of the two proposed modifications on challenging, multi-lingual real world data.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Enriching Word Vectors with Subword Information (2016, background): “A first class of such models are recurrent neural networks, applied to language modeling (Mikolov et al., 2012; Sutskever et al., 2011; Graves, 2013; Bojanowski et al., 2015), text normalization (Chrupała, 2014), part-ofspeech tagging (Ling et al.”
Variable Computation in Recurrent Neural Networks (2016, methodology): “We first compare the VCRNN to the regular Elman RNN, as well as to the Conditional RNN of (Bojanowski et al., 2015), which combines two layers running at bit and character level for bit level modeling, or character and word level for character level modeling.”