Deep Temporal Sigmoid Belief Networks for Sequence Modeling

By Zhe Gan, Chunyuan Li, Ricardo Henao, David Edwin Carlson, Lawrence Carin. 2015

91 citations recorded by the source corpus.

Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherited from the previous SBNs in the sequence, and is used to regulate its hidden bias. Scalable learning and inference algorithms are derived by introducing a recognition model that yields fast sampling from the variational posterior. This recognition model is trained jointly with the generative model, by maximizing its variational lower bound on the log-likelihood. Experimental results on bouncing balls, polyphonic music, motion capture, and text streams show that the proposed approach achieves state-of-the-art predictive performance, and has the capacity to synthesize various sequences.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, background): “Finally, several previous works have tried to introduce latent variables into sequence modeling (Bayer & Osendorfer, 2014; Gregor et al., 2015; Chung et al., 2015; Gan et al., 2015; Fraccaro et al., 2016; Chung et al., 2016).”
Deep Kalman Filters (2015, background): “Speciﬁcally, the functions G α , S β , F κ are assumed to be parameterized by deep neural networks.”
Sequential Neural Models with Stochastic Layers (2016, methodology): “In this section the SRNN is evaluated on the modeling of speech and polyphonic music data, as they have shown to be difficult to model without a good representation of the uncertainty in the latent states [3, 7, 11, 12, 15].”
Structured Inference Networks for Nonlinear State Space Models (2016, methodology): “The baselines we compare to in Table 3 also have more complex generative models than the DMM. STORN has edges from x t − 1 to z t given by the recurrence update and TSBN has edges from x t − 1 to z t as well as from x t − 1 to x t .”
Dynamical Variational Autoencoders: A Comprehensive Review (2020, background): “Temporal models with binary observed and latent random variables can be found in (Boulanger- Lewandowski et al., 2012; Gan et al., 2015).”
BLACK BOX VARIATIONAL INFERENCE FOR STATE SPACE MODELS (2015, background): “Recent, related approaches to variational inference in time-series models focus upon the design and learning of rich generative models capable of capturing the statistical structure of large, complex datasets (Gan et al., 2015; Chung et al., 2015).”
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks (2015, background): “Interesting future directions include exploring applications to latent variable models or recurrent neural networks (Gan et al. 2015).”
Deep Poisson Factor Modeling (2015, methodology): “Future directions include working towards alternatives for scaling up inference algorithms based on gradient-based approaches, extending the use of PFA modules in deep architectures to more sophisticated discriminative models, multi-modal tasks with mixed data types, and time series modeling using i”
Scalable Gradients for Stochastic Differential Equations (2020, methodology): “[17], consists of 23 walking sequences of subject 35 partitioned into 16 training, 3 validation, and 4 test sequences.”
Towards Better Representations with Deep/Bayesian Learning (2018, background): “…Chen et al., 2018a; Xu et al., 2017; Wang et al., 2017b; Pu et al., 2018b; Yuan et al., 2017, 2018; Chapfuwa et al., 2018), sigmoid belief networks (Gan et al., 2015b,c,a; Zhang et al., 2016a; Henao et al., 2015a), and their extensions to sequential data generation (Fan et al., 2016b; Zhang et…”
Latent Normalizing Flows for Discrete Sequences (2019, background): “, 2012), TSBN from (Gan et al., 2015), STORN from (Bayer & Osendorfer, 2015), NASMC from (Gu et al.”
Unsupervised Learning of Visual Structure using Predictive Generative Networks (2015, methodology): “There has been much recent success in this domain within natural language processing (NLP) (Graves, 2013; Sutskever et al., 2014) and relatively low-dimensional, real-valued problems such as motion capture (Fragkiadaki et al., 2015; Gan et al., 2015).”