Strongly-Typed Recurrent Neural Networks

By David Balduzzi, Muhammad Ghifary. 2016

66 citations recorded by the source corpus.

Recurrent neural networks are increasing popular models for sequential learning. Unfortunately, although the most effective RNN architectures are perhaps excessively complicated, extensive searches have not found simpler alternatives. This paper imports ideas from physics and functional programming into RNN design to provide guiding principles. From physics, we introduce type constraints, analogous to the constraints that forbids adding meters to seconds. From functional programming, we require that strongly-typed architectures factorize into stateless learnware and state-dependent firmware, reducing the impact of side-effects. The features learned by strongly-typed nets have a simple semantic interpretation via dynamic average-pooling on one-dimensional convolutions. We also show that strongly-typed gradients are better behaved than in classical architectures, and characterize the representational power of strongly-typed nets. Finally, experiments show that, despite being more constrained, strongly-typed architectures achieve lower training and comparable generalization error to classical architectures.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Quasi-Recurrent Neural Networks (2016, methodology): “Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by Balduzzi & Ghifary (2016).”
A Lightweight Recurrent Network for Sequence Modeling (2019, background): “…recent work shows that novel recurrent units derived from weighted ﬁnite state automata are effective without the hidden-to-hidden connection (Balduzzi and Ghifary, 2016; Peng et al., 2018), we empirically observe that including previous hidden states for gates is crucial for model capacity…”
Query-Reduction Networks for Question Answering (2016, methodology): “However, QRN can be distinguished from STRNN in three ways.”
Revisiting Activation Regularization for Language RNNs (2017, background): “Some RNN architectures only allow element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016), removing the recurrent matrix entirely, while others act to restrict the capacity by parameterizing the recurrent matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing…”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, background): “, n-gram filters) within the layer (Balduzzi and Ghifary, 2016; Bradbury et al., 2017), we argue that increasing the depth of the model suffices to retain modeling capacity.”
Training RNNs as Fast as CNNs (2017, background): “nd a fused kernel is created to handle and parallelize both directions. 3 Related Work Many recent work has been dedicated to designing effective networks for sequence processing (Greff et al., 2015; Balduzzi and Ghifary, 2016; Miao et al., 2016; Zoph and Le, 2016; Lee et al., 2017). Our work closel”
Regularizing and Optimizing LSTM Language Models (2017, background): “This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).”