The Inside-Outside Recursive Neural Network model for Dependency Parsing

By Phong Le, Willem H. Zuidema. 2014

73 citations recorded by the source corpus.

We propose the first implementation of an infinite-order generative dependency model. The model is based on a new recursive neural network architecture, the Inside-Outside Recursive Neural Network. This architecture allows information to flow not only bottom-up, as in traditional recursive neural networks, but also topdown. This is achieved by computing content as well as context representations for any constituent, and letting these representations interact. Experimental results on the English section of the Universal Dependency Treebank show that the infinite-order model achieves a perplexity seven times lower than the traditional third-order model using counting, and tends to choose more accurate parses in k-best lists. In addition, reranking with this model achieves state-of-the-art unlabelled attachment scores and unlabelled exact match scores.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling (2017, methodology): “One of the most popular and attractive approaches is to use tree-structured recursive neural networks (Socher et al., 2013; Le and Zuidema, 2014; Dyer et al., 2015), including stacking them on top of a sequential BiLSTM (Miwa and Bansal, 2016).”
Transition-Based Dependency Parsing with Stack Long Short-Term Memory (2015, methodology): “Neural networks have also been used to learn representations for use in chart parsing (Henderson, 2004; Titov and Henderson, 2007; Socher et al., 2013a; Le and Zuidema, 2014).”
Neuro-Symbolic Program Synthesis (2016, methodology): “The Inside-Outside Recursive Neural Network (Le & Zuidema, 2014) in particular is most similar to the R3NN, where they generate a parse tree incrementally by using global leaf-level representations to determine which expansions in the parse tree to take next.”
Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations (2016, background): “The different systems and the numbers reported from them are taken from: ZhangNivre11: (Zhang and Nivre, 2011); Martins13: (Martins et al., 2013); Weiss15 (Weiss et al., 2015); Pei15: (Pei et al., 2015); Dyer15 (Dyer et al., 2015); Ballesteros16 (Ballesteros et al., 2016); LeZuidema14 (Le and Zuidem”
A Primer on Neural Network Models for Natural Language Processing (2015, methodology): “Works using the reranking approach include (Socher et al., 2013; Auli et al., 2013; Le & Zuidema, 2014; Zhu et al., 2015a) MEMM and hybrid approaches Other formulations are, of course, also possible.”
Easy-First Dependency Parsing with Hierarchical Tree LSTMs (2016, methodology): “…2010; Stenetorp, 2013; Socher et al., 2013a); and parse-reranking approaches that first produced a k-best list of parses using a traditional parsing technique, and then scores the trees based on a recursive vector encoding of each node (Le and Zuidema, 2014; Le and Zuidema, 2015; Zhu et al., 2015).”
Neural Network Methods for Natural Language Processing (2017, background): “2Le and Zuidema [2014] extend the RecNN definition such that each node has, in addition to its inside state vector, also an outside state vector representing the entire structure around the subtree rooted at that node.”
SCORES (2018, methodology): “This is achieved with a bottom-up encoding for structural information aggregation, followed by a topdown decoding for contextual information propagation [Le and Zuidema 2014].”
Transferring Inductive Biases through Knowledge Distillation (2020, methodology): “The recursion in a model can be implemented in different ways, like in Recurrent Neural Networks [9], Recursive Neural Networks [34, 22] and Universal Transformers [6, 15].”
Graph Convolutional Networks With Argument-Aware Pooling for Event Detection (2018, background): “2015), dependency parsing (Le and Zuidema 2014; Dyer et al. 2015), relation extraction (Li et al.”
Global Belief Recursive Neural Networks (2014, unknown): “Bidirectional recursive models [12, 15], developed in parallel with ours, extend the definition of the recursive neural network by adding a backward propagation step, where information also flows from the tree root back to the leaves.”
An Effective Neural Network Model for Graph-based Dependency Parsing (2015, methodology): “• Unlike other neural network based models (Chen et al., 2014; Le and Zuidema, 2014) where an additional parser is needed for either extracting features (Chen et al., 2014) or generating k-best list for reranking (Le and Zuidema, 2014), both training and decoding in our model are performed based on…”
Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders (2019, background): “Neural Inside-Outside Parsers The Inside-Outside Recursive Neural Network (IORNN) (Le and Zuidema, 2014) is closest to ours.”
Neural CRF Parsing (2015, methodology): “…inference by making sequential decisions (Henderson, 2003; Chen and Manning, 2014; Tsuboi, 2014) or by doing reranking (Socher et al., 2013; Le and Zuidema, 2014); by contrast, our framework permits exact inference via CKY, since the model’s structured interactions are purely discrete and…”
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser (2016, methodology): “…a strong LSTM FOG parser trained using the conventional Hamming cost function, (ii) recently published strong LSTM FOG parsers (Kiperwasser and Goldberg, 2016; Wang and Chang, 2016), and (iii) many higher-order graph-based parsers (Koo and Collins, 2010; Martins et al., 2013; Le and Zuidema, 2014).”
Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation (2015, background): “Recently, recursive neural networks have been proposed for syntactic parsing (Socher et al., 2010; Le and Zuidema, 2014).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, unknown): “As far as the authors are aware, the ﬁrst neural model to achieve state-of-the-art performance in parsing was that of Le and Zuidema [98].”
A Bayesian Model for Generative Transition-based Dependency Parsing (2015, background): “Le and Zuidema (2014) proposed a recursive neural network generative model over dependency trees.”