Stack-propagation: Improved Representation Learning for Syntax

By David Weiss. 2016

87 citations recorded by the source corpus.

Traditional syntax models typically leverage part-of-speech (POS) information by constructing features from hand-tuned templates. We demonstrate that a better approach is to utilize POS tags as a regularizer of learned representations. We propose a simple method for learning a stacked pipeline of models which we call “stack-propagation”. We apply this to dependency parsing and tagging, where we use the hidden layer of the tagger network as a representation of the input tokens for the parser. At test time, our parser does not require predicted POS tags. On 19 languages from the Universal Dependencies, our method is 1.3% (absolute) more accurate than a state-of-the-art graph-based approach and 2.7% more accurate than the most comparable greedy model.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “For NLP, most work has focused on a small number of closely related tasks (Luong et al., 2016; Zhang and Weiss, 2016; Søgaard and Goldberg, 2016; Peng et al., 2017).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “To jointly learn the two-stage model, a stack propagation method was applied (Zhang and Weiss, 2016).”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, positive): “MTL (Caruana, 1993) is popular in NLP, and others have proposed MTL models which incorporate subsets of the tasks we do (Collobert et al., 2011; Zhang and Weiss, 2016; Hashimoto et al., 2017; Peng et al., 2017; Swayamdipta et al., 2017), and we build off work that investigates where and when to comb”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, background): “Several approaches integrate supervised feedback from each task at levels consistent with such a hierarchy (Hashimoto et al., 2016; Toshniwal et al., 2017; Zhang and Weiss, 2016) (Figure 1c).”
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing (2019, background): “Features are CNN representations of token features and shared across all pipeline models (Kiperwasser and Goldberg, 2016; Zhang and Weiss, 2016).”
Multi-Task Learning for Sequence Tagging: An Empirical Study (2018, methodology): “Exceptions to this include the work of (Collobert et al., 2011), which considers four tasks: POS , CHUNK , NER , and SRL ; (Raganato et al., 2017), which considers three: word sense disambiguation with POS and coarse-grained semantic tagging based on WordNet lexicographer ﬁles; (Hashimoto et al., 20”
Syntactic Scaffolds for Semantic Structures (2018, methodology): “Similar to our work, there have been multitask models that use partof-speech tagging as T2, with transition-based dependency parsing (Zhang and Weiss, 2016) and CCG supertagging (Søgaard and Goldberg, 2016) as T1.”
Evolutionary architecture search for deep multitask networks (2018, background): “epths for different tasks. Deep MTL has extended these ideas to domains where deep learning thrives, including vision [2, 17, 26, 31, 33, 35, 49, 51], speech [11,12,17,39,48],naturallanguageprocessing[4,6,9,17,24,27,50], and reinforcement learning [5, 15, 47]. The key design decision in constructing”
Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation (2017, methodology): “Figure 2: Applied stack propagation (Zhang and Weiss, 2016) to Predictor-Estimator architecture by alternating stochastic updates.”