Deep Biaffine Attention for Neural Dependency Parsing

By Timothy Dozat, Christopher D. Manning. 2016

1,337 citations recorded by the source corpus.

This paper builds off recent work from Kiperwasser & Goldberg (2016) using neural attention in a simple graph-based dependency parser. We use a larger but more thoroughly regularized parser than other recent BiLSTM-based approaches, with biaffine classifiers to predict arcs and labels. Our parser gets state of the art or near state of the art performance on standard treebanks for six different languages, achieving 95.7% UAS and 94.1% LAS on the most popular English PTB dataset. This makes it the highest-performing graph-based parser on this benchmark---outperforming Kiperwasser Goldberg (2016) by 1.8% and 2.2%---and comparable to the highest performing transition-based parser (Kuncoro et al., 2016), which achieves 95.8% UAS and 94.6% LAS. We also show which hyperparameter choices had a significant effect on parsing accuracy, allowing us to achieve large gains over other graph-based approaches.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “(Chiu and Nichols, 2016) 91.62 ± 0.33 86.28 ± 0.26 ID-CNN-CRF (Strubell et al., 2017) 90.65 ± 0.15 86.84 ± 0.19 Tri-Trained LSTM (Lewis et al., 2016) 94.7 Shortcut LSTM (Wu et al., 2017) 95.08 97.53 JMT* (Hashimoto et al., 2017) 95.77 97.55 94.67 92.90 LM-LSTM-CNN-CRF (Liu et al., 2017) 95.96 ± 0.08”
Universal Language Model Fine-tuning for Text Classification (2018, neutral): “99 , similar to (Dozat and Manning, 2017).”
SciBERT: A Pretrained Language Model for Scientific Text (2019, methodology): “The SOTA result for GENIA is in Nguyen and Verspoor (2019) which uses the model from Dozat and Manning (2017) with part-of-speech (POS) features, which we do not use.”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “To evaluate such syntactic sensitivity in VisualBERT, we ﬁrst parse all sentences in Flickr30K us-ing AllenNLP’s dependency parser (Dozat & Manning, 2017; Gardner et al., 2018).”
Energy and Policy Considerations for Deep Learning in NLP (2019, neutral): “Advances in techniques and hardware for training deep neural networks have recently enabled impressive accuracy improvements across many fundamental NLP tasks (Bahdanau et al., 2015; Luong et al., 2015; Dozat and Manning, 2017; Vaswani et al., 2017), with the most computationally-hungry models obtai”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “This approach is related to Dozat and Manning (2017) and Zhang et al. (2017), where the main difference is that our model works on a multi-task framework.”
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology (2019, methodology): “For each of the four languages, we parsed one million sentences from Wikipedia (May 2018 dump) using Dozat and Manning (2016)’s parser and extracted taggings and lemmata using the method of Müller et al. (2015).”
Sentence-State LSTM for Text Representation (2018, methodology): “Bi-directional extensions have become a standard conﬁguration for achieving state-of-the-art accuracies among various tasks (Wen et al., 2015; Ma and Hovy, 2016; Dozat and Manning, 2017).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, neutral): “In each transformer block, hi+1· = Skip(FF, Skip(MHSA, h i ·)) Skip(f, h) = LN(h+ Dropout(f(h))) FF(h) = GELU(hW⊤1 + b1)W ⊤ 2 + b2 where GELU is an element-wise activation function (Hendrycks and Gimpel, 2016).”
What Does BERT with Vision Look At? (2020, methodology): “For evaluation, we parse all sentences in the validation set of Flickr30K using AllenNLP (Dozat and Manning, 2017; Gardner et al., 2018) and use the parser output as the gold parsing annotation.”
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages (2020, methodology): “For consistency among universal POS (UPOS), treebank-speciﬁc POS (XPOS), and UFeats, we adopt the biafﬁne scoring mechanism from Dozat and Manning (2017) to condition XPOS and UFeats prediction on that of UPOS.”
Evaluating NLP Models via Contrast Sets (2020, background): “The parser that we use achieves 95.7% unlabeled attachment score on the English Penn Treebank (Dozat and Manning, 2017).”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, background): “• The parser-specific bidirectional LSTM layer, whose output is then passed to a bi-affine attention layer (Dozat and Manning, 2017) pro-”