Recurrent Additive Networks

By Kenton C. T. Lee, Omer Levy, Luke S. Zettlemoyer. 2017

38 citations recorded by the source corpus.

We introduce recurrent additive networks (RANs), a new gated RNN which is distinguished by the use of purely additive latent state updates. At every time step, the new state is computed as a gated component-wise sum of the input and the previous state, without any of the non-linearities commonly used in RNN transition dynamics. We formally show that RAN states are weighted sums of the input vectors, and that the gates only contribute to computing the weights of these sums. Despite this relatively simple functional form, experiments demonstrate that RANs perform on par with LSTMs on benchmark language modeling problems. This result shows that many of the non-linear computations in LSTMs and related networks are not essential, at least for the problems we consider, and suggests that the gates are doing more of the computational work than previously understood.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Training RNNs as Fast as CNNs (2017, methodology): “Many recent work has been dedicated to designing effective networks for sequence processing (Greff et al., 2015; Balduzzi and Ghifary, 2016; Miao et al., 2016; Zoph and Le, 2016; Lee et al., 2017).”
Bridging CNNs, RNNs, and Weighted Finite-State Machines (2018, background): “Recently, several works introduced simpler versions of RNNs, such as recurrent additive networks (Lee et al., 2017) and Quasi-RNNs (Bradbury et al., 2017).”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “…state-of-the-art for a wide range of tasks such as text classiﬁ-cation (Howard and Ruder, 2018), machine translation (Vaswani et al., 2017), semantic role labeling (Zhou and Xu, 2015; He et al., 2017), coreference resolution (Lee et al., 2017a), and semantic parsing (Krishnamurthy et al., 2017).”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, background): “The computation resembles other recurrent networks such as LSTM, GRU and RAN (Lee et al., 2017).”
A Lightweight Recurrent Network for Sequence Modeling (2019, background): “Lee et al. (2017) introduce an additive structure to generate hidden representations with linear transformed inputs directly, though we empirically observe that non-linear activations can stabilize model training.”