Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences

By Daniel Neil, Michael Pfeiffer, Shih-Chii Liu. 2016

496 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. Current RNN models are ill suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors which generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range which require updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

In Time-Series & Forecasting

Citations in the corpus

Recurrent Independent Mechanisms (2019, background): “Computation on demand: There are various architectures (El Hihi & Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016; Neil et al., 2016; Jernite et al., 2016; Krueger et al., 2016) where parts of the LSTM’s hidden state are kept dormant at times.”
Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounders (2019, methodology): “The factor model can be extended to allow for irregularly sampled data by using a PhasedLSTM (Neil et al., 2016).”
Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations (2020, methodology): “Our model can also be modiﬁed to allow for irregular samplings of observations by using a PhasedLSTM (Neil et al., 2016).”
Autoregressive Convolutional Neural Networks for Asynchronous Time Series (2017, methodology): “We evaluate the proposed model on artificially generated datasets, household electric power consumption dataset available from UCI repository (Lichman, 2013), and the financial dataset of bid/ask quotes sent by several market participants active in the credit derivatives market, comparing its perfor”
Dilated Recurrent Neural Networks (2017, background): “Recent attempts have focused on multi-timescale designs, which include clockwork RNNs [12], phased LSTM [17], hierarchical multi-scale RNNs [5], etc.”
Neural Speed Reading with Structural-Jump-LSTM (2019, unknown): “More widely, gated units, such as GRU and LSTM, face problems with long input sequences (Neil et al., 2016).”
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification (2018, methodology): “This results in an event-based architecture which is sig-niﬁcantly faster and more accurate than existing ones [30, 33, 46].”
Malware Detection by Eating a Whole EXE (2017, background): “Other works that have made use of RNNs operating at multiple frequencies have similarly worked on sequences that do not exceed thousands of time steps (Koutnik et al. 2014; Neil, Pfeiffer, and Liu 2016).”
Deep Learning in Spiking Neural Networks (2018, background): “The phased LSTM [220], although not a spiking LSTM, is well suited to process event-driven, asynchronously sampled data, which is a common task for SNNs.”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, background): “A similar behavior was observed for Phased LSTM, where increasing the sparsity of cell updates accelerates training for very long sequences [18].”
Recurrent Neural Networks (2018, background): “, 2017), variable computation (Jernite et al., 2017; Neil et al., 2016) or even external memory (Graves et al.”
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks (2017, background): “Therefore many complicated training skills are required to improve performance [19], [23], [24], such as ﬁxed-amount-proportional reset, lateral inhibition, error normalization, weight/threshold regularization, etc.”