Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

By Miguel Ballesteros, Chris Dyer, Noah A. Smith. 2015

300 citations recorded by the source corpus.

We present extensions to a continuousstate dependency parsing method that makes it applicable to morphologically rich languages. Starting with a highperformance transition-based parser that uses long short-term memory (LSTM) recurrent neural networks to learn representations of the parser state, we replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs. This allows statistical sharing across word forms that are similar on the surface. Experiments for morphologically rich languages show that the parsing model benefits from incorporating the character-based encodings of words.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “There is an increased interest in incorporating character-level inputs to build word embeddings for various NLP problems, including part-of-speech tagging, parsing and language modeling (Ling et al., 2015; Kim et al., 2015; Ballesteros et al., 2015).”
Enriching Word Vectors with Subword Information (2016, neutral): “, 2015) and parsing (Ballesteros et al., 2015). Another family of models are convolutional neural networks trained on characters, which were applied to part-of-speech tagging (dos Santos and Zadrozny, 2014), sentiment analysis (dos Santos and Gatti, 2014), text classification (Zhang et al., 2015) an”
Character-Aware Neural Language Models (2015, background): “Ballesteros, Dyer, and Smith (2015) use an RNN over characters only to train a transitionbased parser, obtaining improvements on many morphologically rich languages.”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, methodology): “The embeddings are obtained via unsupervised learning from co-occurrence information in text, and contain semantic information about the word which are useful for many NLP tasks [4, 5, 6, 7, 8].”
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation (2015, methodology): “Finally, our work has been applied to dependency parsing and found similar improvements over word models in morphologically rich languages (Ballesteros et al., 2015).”
Character-based Neural Machine Translation (2016, background): “for POS tagging (Santos and Zadrozny, 2014), name entity recognition (Santos and aes, 2015), parsing (Ballesteros et al., 2015), normalization (Chrupala, 2014) or learning word representations (Botha and Blunsom, 2014; Chen et al.”
Multilingual Language Processing From Bytes (2015, background): “Recent work has shown that modeling the sequence of characters in each token with an LSTM can more effectively handle rare and unknown words than independent word embeddings (Ling et al., 2015; Ballesteros et al., 2015).”
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models (2016, background): “, 2015a), dependency parsing (Ballesteros et al., 2015), text classification (Zhang et al.”
Charagram: Embedding Words and Sentences via Character n-grams (2016, methodology): “Ballesteros et al. (2015) used this model to represent words for dependency parsing.”
Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers (2016, methodology): “A similar idea was applied to dependency parsing in (Ballesteros et al., 2015).”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “The work of Ballesteros et al. (2015) show that the two-LSTMs encoding of Ling et al. (2015b) is beneficial also for representing words in dependency parsing of morphologically rich languages.”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, background): “Despite its popularity, character-based approach also comes out as an equal competitor (Santos and Zadrozny, 2014; Kim et al., 2016; Ling et al., 2015b; Ling et al., 2015a; Faruqui et al., 2016; Ballesteros et al., 2015) .”
Word Representation Models for Morphologically Rich Languages in Neural Machine Translation (2016, background): “Another strand of research has looked at recurrent architectures, using long-short term memory units (Ling et al., 2015a; Ballesteros et al., 2015) which can capture long orthographic patterns in the character sequence, as well as non-compositionality.”
When is multitask learning effective? Semantic sequence prediction under varying data conditions (2016, methodology): “Character embeddings ~c are incorporated via a hierarchical biLSTM using a sequence bi-LSTM at the lower level (Ballesteros et al., 2015; Plank et al., 2016).”
Recent Advances in Recurrent Neural Networks (2017, background): “In such a controlled structure, not only the controller can push to and pop from the top of the stack in constant time but also an LSTM can maintain a continuous space embedding of the stack contents [58], [59].”
Exploiting Multi-typed Treebanks for Parsing with Deep Multi-task Learning (2016, background): “, 2015) which is further enhanced by modeling characters (Ballesteros et al., 2015).”
Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss (2016, methodology): “, 2015), transition-based dependency parsing (Ballesteros et al., 2015), fine-grained sentiment analysis (Liu et al.”
Stack-propagation: Improved Representation Learning for Syntax (2016, methodology): “We observe a >2% absolute gain in labeled accuracy compared to state-of-the-art, LSTM-based greedy parsers (Ballesteros et al., 2015) and a >1% gain compared to a state-of-the-art, graphbased method (Lei et al., 2014).”