Improving Multi-Step Prediction of Learned Time Series Models

By Arun Venkatraman, Martial Hebert, J. Bagnell. 2015

260 citations recorded by the source corpus.

Most typical statistical and machine learning approaches to time series modeling optimize a single-step prediction error. In multiple-step simulation, the learned model is iteratively applied, feeding through the previous output as its new input. Any such predictor however, inevitably introduces errors, and these compounding errors change the input distribution for future prediction steps, breaking the train-test i.i.d assumption common in supervised learning. We present an approach that reuses training data to make a no-regret learner robust to errors made during multi-step prediction. Our insight is to formulate the problem as imitation learning; the training data serves as a "demonstrator" by providing corrections for the errors made during multi-step prediction. By this reduction of multi-step time series prediction to imitation learning, we establish theoretically a strong performance guarantee on the relation between training error and the multi-step prediction error. We present experimental results of our method, DaD, and show significant improvement over the traditional approach in two notably different domains, dynamic system modeling and video texture prediction.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Signal Processing

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “For ML training, we use the teacher forcing algorithm with the only difference that at each decoding step, we choose with a 25% probability the previously generated token instead of the ground-truth token as the decoder input token y t − 1 , which reduces exposure bias (Venkatraman et al., 2015).”
Learning Latent Dynamics for Planning from Pixels (2018, neutral): “Venkatraman et al. (2015) take an imitation learning approach.”
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (2015, methodology): “In comparison to SEARN and related ideas [10, 11], our proposed approach is completely online: a single model is trained and the policy slowly evolves during training, instead of a batch approach, which makes it much faster to train3 Furthermore, SEARN has been proposed in the context of reinforceme”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “This can be achived through maximum likelihood estimation or other techniques from generative and dynamics modeling [61, 62, 63].”
Dual Policy Iteration (2018, methodology): “We believe the dependency on horizon can be brought down by leveraging system identification methods focusing on multi-step prediction [35, 36].”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “Scheduled Sampling (SS) (Bengio et al., 2015) and Data as Demonstrator (DaD) (Venkatraman et al., 2015) are similar techniques that aim to alleviate the mismatch between training and inference prefix distributions.”
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction (2017, methodology): “In contrast with reinforcement learning methods, well-known imitation learning (IL) and sequential prediction algorithms such as SEARN (Daum´e III et al., 2009), DaD (Venkatraman et al., 2015), AggreVaTe (Ross & Bagnell, 2014), and LOLS (Chang et al., 2015b) reduce the sequence prediction problem to”
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood (2017, background): “It is tempting to group our approach with sequence learning methods which interpolate between supervised learning and reinforcement learning (Ranzato et al., 2015; Venkatraman et al., 2015; Ross et al., 2011; Norouzi et al., 2016; Bengio et al., 2015; Levine, 2014).”
A New GAN-based End-to-End TTS Training Algorithm (2019, methodology): “The widely used algorithm is to feed-back the generated data in training with a sampling strategy, e.g. data as demonstrator (DAD) [14] and scheduled sampling (SS) [15].”
Information-Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving (2017, background): “Minimizing multistep prediction error is technically the correct objective, but considerably more difficult as the problem becomes nonconvex [44].”
Cold-Start Reinforcement Learning with Softmax Policy Gradient (2017, methodology): “Although widely used in many different applications, MLE estimation for sequence learning suffers from the exposure-bias problem [24, 19].”
Improved Learning of Dynamics Models for Control (2016, methodology): “We leverage Data As Demonstrator (DaD) [10] to generate synthetic training examples for improving the learned dynamics model’s multistep prediction capabilities.”
Information theoretic MPC for model-based reinforcement learning (2017, background): “In the worst case, compounding multi-step errors can grow exponentially [24].”
Learning to Filter with Predictive State Inference Machines (2015, methodology): “Data as Demonstrator (DaD) (Venkatraman et al., 2015) applies the Inference Machine idea to fully observable Markov chains, and directly optimizes the open-loop forward prediction accuracy.”
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation (2019, methodology): “For time series modeling, the Data as Demonstrator algorithm (Venkatraman et al., 2015) derives the expert policy directly from the reference sequences which are aligned with the sampled sequences at each time step.”