Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations

By Eliyahu Kiperwasser, Yoav Goldberg. 2016

688 citations recorded by the source corpus.

We present a simple and effective scheme for dependency parsing which is based on bidirectional-LSTMs (BiLSTMs). Each sentence token is associated with a BiLSTM vector representing the token in its sentential context, and feature vectors are constructed by concatenating a few BiLSTM vectors. The BiLSTM is trained jointly with the parser objective, resulting in very effective feature extractors for parsing. We demonstrate the effectiveness of the approach by applying it to a greedy transition-based parser as well as to a globally optimized graph-based parser. The resulting parsers have very simple architectures, and match or surpass the state-of-the-art accuracies on English and Chinese.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Structured Attention Networks (2017, methodology): “A.1 S YNTACTIC A TTENTION The syntactic attention layer (for tree transduction and natural language inference) is similar to the ﬁrst-order graph-based dependency parser of Kipperwasser & Goldberg (2016).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2016), syntactic parsing (Kiperwasser and Goldberg, 2016), machine translation (MT) (Bahdanau et al.”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, positive): “, 2012), parsing (Vinyals et al., 2015; Kiperwasser and Goldberg, 2016; Dyer et al., 2016), machine translation (Bahdanau et al.”
Colorless Green Recurrent Networks Dream Hierarchically (2018, background): “success of RNNs as feature extractors for syntactic parsing (e.g., Cross and Huang, 2016; Kiperwasser and Goldberg, 2016; Zhang et al., 2017).”
Deep Biaffine Attention for Neural Dependency Parsing (2016, methodology): “Kiperwasser & Goldberg (2016) present a neural graph-based parser (in addition to a transition-based one) that uses the same kind of attention mechanism as Bahdanau et al. (2014) for machine translation.”
What Do Recurrent Neural Network Grammars Learn About Syntax? (2016, methodology): “As a by-product of our investigation, a variant of the RNNG without ensembling achieved the best reported supervised phrase-structure parsing (93.6 F 1 ; English PTB) and, through conversion, dependency parsing (95.8 UAS, 94.6 LAS; PTB SD).”
When is multitask learning effective? Semantic sequence prediction under varying data conditions (2016, methodology): “Bi-LSTM have recently successfully been used for a variety of tasks (Col-lobert et al., 2011; Huang et al., 2015; Dyer et al., 2015; Ballesteros et al., 2015; Kiperwasser and Goldberg, 2016; Liu et al., 2015; Plank et al., 2016).”
Learning What’s Easy: Fully Differentiable Neural Easy-First Taggers (2017, methodology): “Usually, gradients are not backpropagated over this combinatorial latent space (Kiperwasser and Goldberg, 2016), or a separate model is used to determine the easiest next move (Clark and Manning, 2016).”
Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss (2016, methodology): “, 2015), transition-based dependency parsing (Ballesteros et al., 2015; Kiperwasser and Goldberg, 2016), fine-grained sentiment analysis (Liu et al.”
Stack-Pointer Networks for Dependency Parsing (2018, background): “Incorporating this global search algorithm with distributed representations learned from neural networks, neural graph-based parsers (Kiperwasser and Goldberg, 2016; Wang and Chang, 2016; Kuncoro et al., 2016; Dozat and Manning, 2017) have achieved the state-of-the-art accuracies on a number of tree”
Multi-task Domain Adaptation for Sequence Tagging (2016, neutral): “…in NLP and have achieved impressive performance in problems such as sequence tagging (Lample et al., 2016; Yang et al., 2016; Ma and Hovy, 2016), relation classiﬁcation (Xu et al., 2015; Zhang et al., 2015), and syntactic parsing (Kiperwasser and Goldberg, 2016; Cross and Huang, 2016).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, methodology): “Bidirectional-LSTMs have been used in dependency parsers for feature representation [89].”
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, methodology): “, bidirectional LSTMs) has proven successful in structured prediction, such as syntactic and semantic parsing (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2017, 2018; Kasai et al., 2018).”