Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

By Yikang Shen, Shawn Tan, Alessandro Sordoni, Aaron C. Courville. International Conference on Learning Representations, 2018

349 citations recorded by the source corpus.

Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/15d6f3d815d0ff176fafb14a3f46e5723ebac723

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders (2019, methodology): “On the MultiNLI dataset, PRPN-LM is the top performing model without using the +PP heuristic while DIORA matches PRPN-UP (Table 2.”
Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT (2020, methodology): “Take SBAR (clause introduced by a subordinating conjunction) for example, BERT achieves an accuracy of 51.9%, which is about 3.4 times higher than that of ON-LSTM.”
Integrating Tree Path in Transformer for Code Representation (2021, background): “Several works [Shen et al., 2018, Shiv and Quirk, 2019] proposed new architectures for modeling tree structures, including in the domain of natural languages processing [Wang et al., 2019] and code representation learning [Hellendoorn et al., 2019, Zügner et al., 2021].”
Unsupervised Recurrent Neural Network Grammars (2019, methodology): “0 before feeding it to the CRF. 15 Using the code from https://github.com/yikangshen/ PRPN, we tuned model size, initialization, dropout, learning rate, and use of batch normalization.”
Learning Syntax from Naturally-Occurring Bracketings (2021, background): “The models are usually generative and learn from (re)constructing sentences based on induced structures (Shen et al., 2018, 2019; Drozdov et al., 2019; Kim et al., 2019a,b).”
Language-Agnostic Representation Learning of Source Code from Structure and Context (2021, background): “Notably, tree structures have shown promising results and inspired new architectures (Shen et al., 2019), including in the domain of source code (Fernandes et al., 2019), where the underlying syntax is directly available.”
Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization (2019, background): “Recently, Shen et al. (2018) showed that recurrent networks can learn to recognize logical inference relations if they are extended with a bias towards modelling hierarchical structures.”
Visually Grounded Compound PCFGs (2020, methodology): “We use the same data preprocessing 4 as in Shen et al. (2019) and Kim et al. (2019a), where punctuation is removed from all data, and the top 10,000 frequent words in training sentences are kept as the vocabulary.”
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future (2019, background): “Syntactic distances have been proved effective in constituent parsing tasks (Shen et al., 2018a).”
Visually Grounded Neural Syntax Acquisition (2019, positive): “Following the original paper (Shen et al., 2019), we set the number of layers to be 3, and report the constituency parse tree with respect to the gate values output by the second layer of ON-LSTM.”
Modeling Recurrence for Transformer (2019, methodology): “BLEU Existing NMT systems (Vaswani et al., 2017) (Shaw et al., 2018), “D I SAN”: directional SAN (Shen et al., 2018), “RNN Encoder”: combining SAN and RNN encoders with multi-column strategy (Chen et al., 2018).”
Explicitly Modeling Adaptive Depths for Transformer (2020, background): “Another group of works explore to conduct conditional computation inside the dimension of neural networks, (Jernite et al., 2017; Shen et al., 2019), instead of activating partial layers of model, e.g., adaptive depths in our method.”
Compound Probabilistic Context-Free Grammars for Grammar Induction (2019, result): “Given the above, we mainly compare our approach against two recent, strong baselines with open source code: Parsing Predict Reading Network (PRPN) 10 (Shen et al., 2018) and Ordered Neurons (ON) 11 (Shen et al., 2019).”
Neural Machine Translation: A Review (2019, background): “[577] who rearranged neurons in an LSTM network to resemble”
A Comprehensive Survey of Grammar Error Correction (2020, methodology): “To be more specific, the generated ungrammatical sentence 𝑦̃ can be added to training set only when 𝑦̃ and the error-free original sentence 𝑦 satisfy the following inequation: where The variable σ is learned on the development set with size |𝑁| : Htut and Tetreault compared several neural transl”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “(43) Htut and Tetreault compared several neural translation models that could be used on artificial error generation: multi-layer CNN, Transformer, PRPN [119], and ON-LSTM [120].”
Refining Targeted Syntactic Evaluation of Language Models (2021, background): “, Ordered Neurons (Shen et al., 2019) and Recurrent Neural Network Grammars (Dyer et al.”
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020, positive): “Supported models include: ELMo (Peters et al., 2018), GPT (Radford, 2018), BERT (Devlin et al., 2019), XLM (Con-neau and Lample, 2019), GPT-2 (Radford et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019b), and ALBERT (Lan et al., 2019). jiant also supports the from-scratch training o”