Character-Aware Neural Language Models

By Yoon Kim, Yacine Jernite, David A. Sontag, Alexander M. Rush. 2015

1,725 citations recorded by the source corpus.

We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway net work over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “Recent efforts on small scale language modeling have used CNN character embeddings for the input embeddings (Kim et al., 2015).”
Neural Machine Translation of Rare Words with Subword Units (2015, background): “Various techniques have been proposed to produce ﬁxed-length continuous word vectors based on characters or morphemes (Luong et al., 2013; Botha and Blunsom, 2014; Ling et al., 2015a; Kim et al., 2015).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “Similar to Kim et al. (2015), our character-to-word encoder is a ﬁve-layer sub-module that ﬁrst embeds single characters with an embedding layer then passes them through 2048 character n-gram CNN ﬁlters with max pooling, two highway layers (Srivastava et al., 2015), and a linear projection down to…”
Deep Contextualized Word Representations (2018, methodology): “The pre-trained biLMs in this paper are similar to the architectures in J´ozefowicz et al. (2016) and Kim et al. (2015), but modiﬁed to support joint training of both directions and add a residual connection between LSTM layers.”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “These encoders are typically implemented as convolutional (Kim, 2014), recursive (Socher et al., 2013), or recurrent neural networks (Mikolov et al., 2010) operating over a sentence’s words or characters (Zhang et al., 2015; Kim et al., 2016).”
Sequence-Level Knowledge Distillation (2016, background): “• There have been promising recent results on eliminating word embeddings completely and obtaining word representations directly from characters with character composition models, which have many fewer parameters than word embedding lookup tables (Ling et al., 2015a; Kim et al., 2016; Ling et al., 2”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “Position is encoded via ﬁxed sinusoidal position embeddings and we use a character CNN encoding of the input tokens for word-based models (Kim et al., 2016).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “, 2003), characters for the input (Kim et al., 2016), or also the output (Merity et al.”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, neutral): “8 Model #Param Validation Test Mikolov & Zweig (2012) – RNN-LDA + KN-5 + cache 9Mz - 92.0 Zaremba et al. (2014) – LSTM 20M 86.2 82.7 Gal & Ghahramani (2016) – Variational LSTM (MC) 20M - 78.6 Kim et al. (2016) – CharCNN 19M - 78.9 Merity et al. (2016) – Pointer Sentinel-LSTM 21M 72.4 70.9 Gr”
Pointer Sentinel Mixture Models (2016, background): “Parameter numbers with ‡ are estimates based upon our understanding of the model and with reference to Kim et al. (2016).”
What do Neural Machine Translation Models Learn about Morphology? (2017, methodology): “For a character-based model we adopt a convolutional neural network (CNN) over character embeddings that is also learned during training (Kim et al., 2015); see appendix A.1 for speciﬁc settings.”
Bottom-Up Abstractive Summarization (2018, background): “, 2018) which uses a character-aware token embedding (Kim et al., 2016) followed by two bidirectional LSTM layers h((1)) i and h (2) i .”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, background): “The convolutions are followed by a Tanh nonlinearity and max-pooling over the length over the word (Kim et al., 2015).”
Recurrent Highway Networks (2016, background): “, 2016) and language modeling (Kim et al., 2015; Jozefowicz et al., 2016), and their variants called Residual networks have been widely useful for many computer vision problems (He et al.”