Learning Dynamic Generator Model by Alternating Back-Propagation Through Time

By Jianwen Xie, Ruiqi Gao, Zilong Zheng, Song-Chun Zhu, Ying Nian Wu. 2018

48 citations recorded by the source corpus.

This paper studies the dynamic generator model for spatialtemporal processes such as dynamic textures and action sequences in video data. In this model, each time frame of the video sequence is generated by a generator model, which is a non-linear transformation of a latent state vector, where the non-linear transformation is parametrized by a top-down neural network. The sequence of latent state vectors follows a non-linear auto-regressive model, where the state vector of the next frame is a non-linear transformation of the state vector of the current frame as well as an independent noise vector that provides randomness in the transition. The non-linear transformation of this transition model can be parametrized by a feedforward neural network. We show that this model can be learned by an alternating back-propagation through time algorithm that iteratively samples the noise vectors and updates the parameters in the transition model and the generator model. We show that our training method can learn realistic models for dynamic textures and action patterns.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

In Optimization & Training

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “Nevertheless, the dynamic generator [63] and Fig.”
VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation (2019, background): “Much of this research on deterministic models focused on architectural changes, such as predicting high-level structure (Villegas et al., 2017b), energy-based models (Xie et al., 2017), generative cooperative nets (Xie et al., 2020), ABPTT (Xie et al., 2019), incorporating pixel transformations (Fin”
Stochastic Image-to-Video Synthesis using cINNs (2021, result): “As described in the main paper, DG [31] is directly optimized on test samples, thus overfitting directly to the test distribution.”