Z-Forcing: Training Stochastic Recurrent Networks

By Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, Yoshua Bengio. 2017

198 citations recorded by the source corpus.

Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We unify successful ideas from recently proposed architectures into a stochastic recurrent model: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortized variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. We found this strategy to perform better than alternative approaches such as KL annealing. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and Blizzard and competitive performance on sequential MNIST. Finally, we apply our model to language modeling on the IMDB dataset where the auxiliary cost helps in learning interpretable latent variables. Source Code: \url{this https URL}

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Lifelong Generative Modeling (2017, background): “This is similar to the posterior collapse phenomenon which has received a lot of recent interest within the VAE community [99, 40].”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, methodology): “[18] use an auxiliary reconstruction loss to avoid posterior collapse in recurrent models.”
Preventing Posterior Collapse with delta-VAEs (2019, methodology): “Most common approaches either change the objective (Higgins et al., 2017; Alemi et al., 2017; Zhao et al., 2017; Chen et al., 2016; Lucas & Verbeek, 2017; Goyal et al., 2017), or weaken the decoder (Bowman et al.”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, background): “Other attempts to address posterior collapse include proposing new regularizers (Zhao et al., 2017; Goyal et al., 2017; Tolstikhin et al., 2018; Phuong et al., 2018), deploying less powerful de-coders (Yang et al., 2017; Semeniuta et al., 2017), using lossy input (Chen et al., 2017), utilizing…”
Semi-Amortized Variational Autoencoders (2018, background): “Note that unlike to sequential VAEs that employ different latent variables at each time step (Chung et al., 2015; Fraccaro et al., 2016; Krishnan et al., 2017; Serban et al., 2017; Goyal et al., 2017a), in this work we focus on modeling the entire sequence with a global latent variable.”
Latent Alignment and Variational Attention (2018, background): “Of particular interest are those that augment an RNN with latent variables (typically Gaussian) at each time step [17, 22, 66, 23, 40] and those that incorporate latent variables into sequence-to-sequence models [84, 7, 70, 64].”
A General Method for Amortizing Variational Filtering (2018, methodology): “Amortized variational inference [27, 37] has enabled many recently proposed probabilistic deep dynamical latent variable models, with applications to video [42, 26, 43, 23, 15, 11, 3, 9, 29, 19], speech [7, 12, 17, 22, 29], handwriting [7], music [12], etc.”
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning (2019, background): “Fraccaro et al. [15] develop SRNN, Chung et al. [8] devise VRNN, and Goyal et al. [17] propose Z-forcing.”
Generalization in Generation: A closer look at Exposure Bias (2019, methodology): “…introducing uncertainty inside the model is often motivated as a regularizing technique in Bay-seian machine learning (Murphy, 2012) and has been applied extensively to latent sequence models (M. Ziegler and M. Rush, 2019; Schmidt and Hofmann, 2018; Goyal et al., 2017; Bayer and Os-endorfer, 2014).”
Towards Text Generation with Adversarially Learned Neural Outlines (2018, background): “However, VAEs for text are notoriously hard to train when combined with powerful autoregressive decoders [5, 18, 47].”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, unknown): “Three widely used increasing functions for f are linear (Fraccaro et al., 2016; Goyal et al., 2017), Sigmoid (Bowman et al., 2015) and Consine (Lai et al., 2018).”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, methodology): “However, in practice it is difﬁcult for latent variables to capture higher level representation in the presence of a strong autoregressive model as shown in Gulrajani et al. (2016); Goyal et al. (2017); Guu et al. (2018).”
Generating Multi-Agent Trajectories using Programmatic Weak Supervision (2018, background): “Recent work in sequential generative models address this issue by injecting stochastic latent variables into the model and optimizing using amortized variational inference to learn the latent variables (Fraccaro et al., 2016; Goyal et al., 2017).”
Controllable Paraphrase Generation with a Syntactic Exemplar (2019, methodology): “Some recent work used multi-task learning as a way of improving the quality or disentanglement of learned representations (Zhao et al., 2017; Goyal et al., 2017; Du et al., 2018; John et al., 2018).”
Multiple Futures Prediction (2019, background): “Connections to other Stochastic RNNs Various stochastic recurrent models in existing literature have been proposed: DRAW [20], STORN [3], VRNN [11], SRNN [18], Z-forcing [19], Graph-VRNN [31].”
Learning Longer-term Dependencies in RNNs with Auxiliary Losses (2018, background): “In variational inference setting, (GOYAL et al., 2017) also propose reconstruction of the states from a backward running recurrent network as an auxiliary cost to help with training on long sequences.”
Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning (2019, methodology): “Data We use two datasets, Yelp restaurant reviews and Amazon product reviews (He and McAuley 2016)2, which are commonly used in prior works too (Shen et al. 2017; Fu et al. 2018; Li et al. 2018; Prabhumoye et al. 2018).”
Latent Normalizing Flows for Discrete Sequences (2019, methodology): “In these models, the KL term accounts for less than 5% of the ELBO (Bowman et al., 2016; Kim et al., 2018; Xu & Durrett, 2018), or less than 30% of the ELBO when using a specially designed auxiliary loss (Goyal et al., 2017).”