Globally Normalized Transition-Based Neural Networks

By D. Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, Michael Collins. 2016

577 citations recorded by the source corpus.

We introduce a globally normalized transition-based neural network model that achieves state-of-the-art part-of-speech tagging, dependency parsing and sentence compression results. Our model is a simple feed-forward neural network that operates on a task-specific transition system, yet achieves comparable or better accuracies than recurrent models. We discuss the importance of global as opposed to local normalization: a key insight is that the label bias problem implies that globally normalized models can be strictly more expressive than locally normalized models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, methodology): “For each Spanish sentence, we classify whether it contains a third-person singular dropped subject or possessive pro-noun using heuristics over morphological analysis, part of speech tags, and syntactic dependency structure (Andor et al., 2016).”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, methodology): “This output embedding is similar to the K-best POS tag feature which has been shown to be effective in syntactic tasks (Andor et al., 2016; Alberti et al., 2015).”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, background): “Global normalization has yielded important benefits on many NLP tasks including parsing and translation [32, 33].”
Demographic Dialectal Variation in Social Media: A Case Study of African-American English (2016, methodology): “We focus on widely used syntactic representations, testing the SyntaxNet neural network-based dependency parser (Andor et al., 2016), 11 which reports state-of-the-art results, including for web corpora.”
Deep Biaffine Attention for Neural Dependency Parsing (2016, background): “2016) augment it with a beam search and a conditional random ﬁeld loss objective to allow the parser to “undo” previous actions once it ﬁnds evidence that they may have been incorrect; and Dyer et al. (2015) and (Kuncoro et al., 2016) instead use LSTMs to represent the stack and buffer, getting…”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “Machine learning models, especially deep neural networks, have been deployed prominently in many real-world applications, such as image classification [28, 49], speech recognition [11, 21], natural language processing [2, 10], and game playing [35, 48].”
Towards Evaluating the Robustness of Neural Networks (2016, methodology): “The L2 distance can remain small when there are many small changes to many pixels.”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “These models have evolved into their deep neural counterparts [29, 1] through the use of recurrent neural networks (RNN) with LSTM [13] cells and attention mechanisms [2].”
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks (2017, methodology): “…issue, namely Label Bias , especially in sequence-to-sequence learning framework, which obstacles the MLE trained models to be optimized globally (Andor et al., 2016; Wiseman & Rush, 2016) To addresses the abovementioned issues in training auto-regressive models, we propose to formulate the…”
What Do Recurrent Neural Network Grammars Learn About Syntax? (2016, neutral): “9 Andor et al. (2016) 94.”
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging (2017, background): “…results comparable, a provided split In recent years, deep neural networks were shown to achieve state-of-the-art performance for a wide range of NLP tasks, including many sequence tagging tasks (Ma and Hovy, 2016), dependency parsing (Andor et al., 2016), and machine translation (Wu et al., 2016).”
Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing (2019, background): “We should note that while accuracy differences may be relatively small, they are within the margins of recent stateof-the-art improvements (Andor et al., 2016) in a task that achieves extremely high accuracy and where further improvement can only be incremen-”
DeepMath - Deep Sequence Models for Premise Selection (2016, neutral): “In natural language processing, deep neural networks have proven useful in language modeling [30], text classiﬁcation [9], sentence pair scoring [4], dependency parsing [2], sentiment analysis [35], conversation modeling [40], and simple question answering [36].”