State Space LSTM Models with Particle MCMC Inference

By Xun Zheng, Manzil Zaheer, Amr Ahmed, Yuan Wang, Eric P. Xing, Alex Smola. 2017

47 citations recorded by the source corpus.

Long Short-Term Memory (LSTM) is one of the most powerful sequence models. Despite the strong performance, however, it lacks the nice interpretability as in state space models. In this paper, we present a way to combine the best of both worlds by introducing State Space LSTM (SSL), which generalizes the earlier work \cite{zaheer2017latent} of combining topic models with LSTM. However, unlike \cite{zaheer2017latent}, we do not make any factorization assumptions in our inference algorithm. We present an efficient sampler based on sequential Monte Carlo (SMC) method that draws from the joint posterior directly. Experimental results confirms the superiority and stability of this SMC inference algorithm on a variety of domains.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Factors for Forecasting (2019, methodology): “This makes the state transition non-linearly determined by the RNN. VRNNs are also used in Latent LSTM Allocation (LLA) (Zaheer et al., 2017) and State-Space LSTM (SSL) (Zheng et al., 2017).”
Deep State Space Models for Time Series Forecasting (2018, methodology): “They were later used in Latent LSTM Allocation [29] and State-Space LSTM [30].”
Attentive State-Space Modeling of Disease Progression (2019, background): “Because our model is non-Markovian, inference of posterior disease states is intractable and cannot be conducted using standard forward-backward routines (e.g., [8, 9, 10]).”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “Inspired by interesting approaches involving continuous-time representation for recurrent neural networks (Voelker et al., 2019; Zheng et al., 2017), Gu et al.”