Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser

By A. Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Noah A. Smith. 2016

77 citations recorded by the source corpus.

We introduce two first-order graph-based dependency parsers achieving a new state of the art. The first is a consensus parser built from an ensemble of independently trained greedy LSTM transition-based parsers with different random initializations. We cast this approach as minimum Bayes risk decoding (under the Hamming cost) and argue that weaker consensus within the ensemble is a useful signal of difficulty or ambiguity. The second parser is a "distillation" of the ensemble into a single model. We train the distillation parser using a structured hinge loss objective with a novel cost that incorporates ensemble uncertainty estimates for each possible attachment, thereby avoiding the intractable cross-entropy computations required by applying standard distillation objectives to problems with structured outputs. The first-order distillation parser matches or surpasses the state of the art on English, Chinese, and German.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Distilling large models into small models (Kim and Rush, 2016; Mou et al., 2016) or ensembles of models into single models (Kuncoro et al., 2016; Liu et al., 2019a) has been shown to improve re-sults for many NLP tasks.”
Knowledge Distillation: A Survey (2020, methodology): “To obtain the lightweight language models with good efficiency and effectiveness, recently knowledge distillation is extensively studied in the field of natural language processing (NLP) and more and more KD methods are proposed in solving the numerous NLP tasks (Liu et al., 2019b; Haidar and Rezagh”
What Do Recurrent Neural Network Grammars Learn About Syntax? (2016, methodology): “The last row reports the performance of the gated attention RNNG model (x4) Model UAS LAS Kiperwasser and Goldberg (2016) 93.9 91.9 Ballesteros et al. (2016) 93.6 91.4 Wang and Chang (2016) 94.1 91.8 Kuncoro et al. (2016) 94.3 92.1 Andor et al. (2016) 94.6 92.8 Dozat and Manning (2016) 95.4 93.8 Cho”
Stronger Baselines for Trustable Results in Neural Machine Translation (2017, background): “Decoding costs can be reduced by using knowledge distillation techniques to train a single, compact model to replicate the output of an ensemble (Hinton et al., 2015; Kuncoro et al., 2016; Kim and Rush, 2016).”
Distilling Neural Networks for Greener and Faster Dependency Parsing (2020, methodology): “Similar and more recent work, used this method of compressing many models into one to achieve state-of-the-art parsing performance (Kuncoro et al., 2016).”
Teacher-Student Networks with Multiple Decoders for Solving Math Word Problem (2020, methodology): “Knowledge distillation was also applied in other tasks such as text classiﬁ-cation [Zagoruyko and Komodakis, 2017], parsing [Kuncoro et al. , 2016] and machine reading comprehension [Hu et al. , 2018].”
Stack-Pointer Networks for Dependency Parsing (2018, methodology): “Incorporating this global search algorithm with distributed representations learned from neural networks, neural graph-based parsers (Kiperwasser and Goldberg, 2016; Wang and Chang, 2016; Kuncoro et al., 2016; Dozat and Manning, 2017) have achieved the state-of-the-art accuracies on a number of tree”
Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System (2019, background): “Distilling complex models into simple models has been shown to improve many NLP tasks to achieve impressive performance [14, 15, 18, 20].”
Head-Driven Phrase Structure Grammar Parsing on Penn Treebank (2019, methodology): “For dependency parsing, following (Dozat and Manning, 2017; Kuncoro et al., 2016; Ma et al., 2018), we report the results without punctuations for both treebanks.”
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, methodology): “Knowledge Distillation Knowledge distillation has been used to improve the performance of small models with the guidance of big models, with applications in natural language processing (Kim and Rush, 2016; Kuncoro et al., 2016; Tan et al., 2019; Clark et al., 2019; Sun et al., 2019), computer vision”
Attention-Guided Answer Distillation for Machine Reading Comprehension (2018, background): “istillation that annotates large-scale unlabelled data for omni-supervised learning. In the ﬁeld of natural language processing, Mou et al. (2016) distill task-speciﬁc knowledge from word embeddings. Kuncoro et al. (2016) propose to learn a single parser from an ensemble of parsers. Kim et al. (2016”
Neural Probabilistic Model for Non-projective MST Parsing (2017, methodology): “Moreover, our model achieves better results than the parser distillation method (Kuncoro et al., 2016) on all the three languages.”
Deep Multitask Learning for Semantic Dependency Parsing (2017, methodology): “Our work is perhaps closest to those who used BiLSTMs to encode inputs (Kiperwasser and Goldberg, 2016; Kuncoro et al., 2016; Wang and Chang, 2016; Dozat and Manning, 2017; Ma and Hovy, 2016).”
Autoregressive Knowledge Distillation through Imitation Learning (2020, unknown): “Subsequent works have used seqKD for non-autoregressive translation models (Gu et al., 2017; Zhou et al., 2019), low-resource settings (Chen et al., 2017), and ensemble distillation with multiple teachers (Kuncoro et al., 2016; Tan et al., 2019).”
Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling (2016, methodology): “However, the cost in model averaging can be alleviated via the distillation methods: learning a single neural network that approximates the results of either a large model or an ensemble of models (Korattikara et al., 2015; Kim and Rush, 2016; Kuncoro et al., 2016).”
Learning Approximate Inference Networks for Structured Prediction (2018, background): “Others have explored knowledge distillation for sequence-to-sequence learning (Kim & Rush, 2016) and parsing (Kuncoro et al., 2016).”
Massively Multilingual Transfer for NER (2019, methodology): “Our work is also related to knowledge distillation from multiple source models applied in parsing (Kuncoro et al., 2016) and machine translation(Kim and Rush, 2016; Johnson et al.”