Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs

By Li Jing, Yichen Shen, T. Dubček, John Peurifoy, S. Skirlo, Yann LeCun, Max Tegmark, Marin Soljačić. 2016

191 citations recorded by the source corpus.

Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data. This approach appears particularly promising for Recurrent Neural Networks (RNNs). In this work, we present a new architecture for implementing an Efficient Unitary Neural Network (EUNNs); its main advantages can be summarized as follows. Firstly, the representation capacity of the unitary space in an EUNN is fully tunable, ranging from a subspace of SU(N) to the entire unitary space. Secondly, the computational complexity for training an EUNN is merely O(1) per parameter. Finally, we test the performance of EUNNs on the standard copying task, the pixel-permuted MNIST digit recognition benchmark as well as the Speech Prediction Test (TIMIT). We find that our architecture significantly outperforms both other state-of-the-art unitary RNNs and the LSTM architecture, in terms of the final performance and/or the wall-clock training speed. EUNNs are thus promising alternatives to RNNs and LSTMs for a wide variety of applications.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “In this case we also compare to the recently-proposed EURNN (Jing et al., 2017), which was highlighted to perform well on this task.”
Continual Learning in Low-rank Orthogonal Subspaces (2020, methodology): “Orthogonal matrices are used in RNNs for avoiding exploding/ vanishing gradient problem [Arjovsky et al., 2016, Wisdom et al., 2016, Jing et al., 2017].”
On orthogonality and learning recurrent networks with long term dependencies (2017, methodology): “Similarly, (Jing et al., 2016) introduce an efficient algorithm to cover a large subset.”
How to Start Training: The Effect of Initialization and Architecture (2018, background): “Activation lengths can also be controlled by constraining weight matrices to be orthogonal or unitary (see e.g. [1, 10, 13, 18, 24]).”
Learning with Hyperspherical Uniformity (2021, background): “There are quite a number of relational regularizations that have been used in neural networks, such as orthogonality regularization [4, 52, 67, 32, 15], unitary constraint [34, 81, 3], decorrelation [67, 16, 85], spectral regularization [86], low-rank regularization [76], angular constraint [84, 44]”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “Various techniques have been proposed to stabilize RNN training (Wisdom et al., 2016; Arjovsky et al., 2015; Jing et al., 2016; Mhammedi et al., 2016; Zhang et al., 2018; Jose et al., 2017; Kusupati et al., 2019) .”
Regularizing and Optimizing LSTM Language Models (2017, background): “This can be done either through restricting the capacity of the matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016) or through element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016).”
Barren plateaus in variational quantum computing (2024, unknown): “One can explicitly verify this by using the Baker–Campbell–Hausdorff formula to re-express the composition of layers as the exponential of a linear combination of the nested commutators of the generators, which by definition belongs in g .”
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics (2019, methodology): “Unitary RNNs (Arjovsky et al., 2016; Jing et al., 2017), similar to LSTMs and GRUs, aim to alleviate the vanishing gradients problem of plain RNNs.”
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations (2019, methodology): “Comparison to related methods Some previous works have examined similar butterfly matrices in numerical algebra or machine learning [34, 19, 30], mainly motivated by trying to parametrize cheap orthogonal matrices.”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “…designed reparameterization of the transition matrix as e.g., with the product of simpler matrices (Arjovsky et al., 2016), Givens rotations (Jing et al., 2017), Householder reﬂections (Mhammedi et al., 2017), or as exponentials of skew-symmetric matrices (Hyland and Rätsch, 2017;…”
Orthogonal Over-Parameterized Training (2020, background): “It is essentially conducting gradient descent on the Stiefel manifold [44, 74, 75, 42, 3, 22, 33].”
Revisiting Activation Regularization for Language RNNs (2017, background): “…only allow element-wise interactions (Balduzzi & Ghifary, 2016; Bradbury et al., 2016; Seo et al., 2016), removing the recurrent matrix entirely, while others act to restrict the capacity by parameterizing the recurrent matrix (Arjovsky et al., 2016; Wisdom et al., 2016; Jing et al., 2016).”
Fast-Slow Recurrent Neural Networks (2017, methodology): “For instance using RNN cells with good long-term memory, like EURNNs [21] or NARX RNNs [26, 8], for the Slow cell might boost the long-term memory of the FS-RNN”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, methodology): “In machine learning, butterﬂy matrices have been use to approximate the Hessian for fast optimization (Mathieu & LeCun, 2014), and to perform fast random projection (Jing et al., 2017; Munkhoeva et al., 2018; Choromanski et al., 2019).”
Convolutional Sequence Modeling Revisited (2018, methodology): “In this case we also compare to the recently-proposed EURNN (Jing et al., 2017), which was highlighted to perform well on this task.”
Wave physics as an analog recurrent neural network (2019, background): “In contrast, the unconstrained dense matrices deﬁning the update relationship of the standard RNN lead to vanishing and ex-ploding gradients, which can pose a major challenge for training traditional RNNs 27 .”
Unifying Orthogonal Monte Carlo Methods (2019, background): “We note that there is a line of work on learning some of these structured contructions (Jing et al., 2017), but in this paper we focus on randomized transformations.”
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (2019, background): “As pointed out in (Jing et al., 2017), this representation, although efficient in memory, does not span the whole space of unitary matrices, giving the model reduced expressiveness.”
Detecting Cyber Attacks in Industrial Control Systems Using Convolutional Neural Networks (2018, background): “in [25] and since then by many others [47] [46] [21]).”