Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

By Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramón Fernandez Astudillo, Silvio Amir, Luís Marujo, Tiago Luís. 2015

648 citations recorded by the source corpus.

We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form‐function relationship in language, our “composed” word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish).

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “Recent efforts on small scale language modeling have used CNN character embeddings for the input embeddings (Kim et al., 2015).”
Deep Contextualized Word Representations (2018, methodology): “Similar to WSD, the biLM representations are competitive with carefully tuned, task speciﬁc biLSTMs (Ling et al., 2015; Ma and Hovy, 2016).”
Sequence-Level Knowledge Distillation (2016, background): “…completely and obtaining word representations directly from characters with character composition models, which have many fewer parameters than word embedding lookup tables (Ling et al., 2015a; Kim et al., 2016; Ling et al., 2015b; Jozefowicz et al., 2016; Costa-Jussa and Fonollosa, 2016).”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “The best result to date has been achieved by Ling et al. (2015), which uses character-based LSTMs. Incorporating the character-based encoders into our JMT model would be an interesting direction, but we have shown that the simple pre-trained character n -gram embeddings lead to the promising result.”
Latent Predictor Networks for Code Generation (2016, methodology): “Inspired by these works, LPNs provide a richer framework by employing attention models (Bahdanau et al., 2014), pointer networks (Vinyals et al., 2015) and character-based embeddings (Ling et al., 2015).”
Hierarchical Attention Networks for Document Classification (2016, methodology): “For a more complete model we could use a GRU to get word vectors directly from characters, similarly to (Ling et al., 2015).”
Neural Architectures for Named Entity Recognition (2016, methodology): “They have been found useful for morphologically rich languages and to handle the outof-vocabulary problem for tasks like part-of-speech tagging and language modeling (Ling et al., 2015b) or dependency parsing (Ballesteros et al., 2015).”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, methodology): “Ling et al. (2015b) indeed states that “[m]uch of the prior information regarding morphology, cognates and rare word translation among others, should be incorporated”.”
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF (2016, methodology): “A similar model has also been applied to Spanish and Portuguese NER (dos Santos et al., 2015) Ling et al. (2015) and Yang et al. (2016) also used BSLTM to compose character embeddings to word’s representation, which is similar to Lample et al. (2016).”
Globally Normalized Transition-Based Neural Networks (2016, methodology): “…variants of long short-term memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) have produced impressive results on some of the classic NLP tasks such as part-of-speech tagging (Ling et al., 2015), syntactic parsing (Vinyals et al., 2015) and semantic role labeling (Zhou and Xu, 2015).”
Character-based Neural Machine Translation (2016, methodology): “Even in MT (Ling et al., 2015b), where authors use the character transformation presented in (Ballesteros et al., 2015; Ling et al., 2015a) both in the source and target.”
Named Entity Recognition with Bidirectional LSTM-CNNs (2015, methodology): “com/p/word2vec/ (5)We used our in-house reimplementation to train word vectors because it uses distributed processing to train much quicker than the publicly-released implementation of word2vec and its performance on the word analogy task was higher than reported by Mikolov et al. (2013). (6)While C”
Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers (2016, methodology): “Based on this intuition, Kim et al. (Kim et al., 2015) and Ling et al. (Ling et al., 2015) proposed to use a character sequence as an alternative to the word-level one-hot vector.”