Model Regularization for Stable Sample Rollouts

By Erik Talvitie. 2014

114 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, background): “For a model of limited capacity and restricted posterior family, the solution to this objective does not necessarily coincide with the best multi-step predictions (Bengio et al., 2015; Talvitie, 2014).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Learning generative models is a very challenging task (Bellemare, Veness, & Bowling, 2013; Bellemare, Veness, & Talvitie, 2014; Chiappa, Racaniere, Wierstra, & Mohamed, 2017; Oh, Guo, Lee, Lewis, & Singh, 2015) and so far, there has been no clear demonstration of successful planning with a learned…”
Recurrent Environment Simulators (2017, methodology): “However, prediction-dependent transitions increase the complexity of the objective function such that alternative schemes are most often used (Talvitie, 2014; Bengio et al., 2015; Oh et al., 2015).”
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, background): “However, a mismatch between the model and the true environment, often unavoidable in practice, can cause this approach to fail (Talvitie, 2014), resulting in policies that do not generalize to the real environment (Jiang et al.”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “Alternatively, a model may also be trained on its own outputs to avoid compounding error from multi-step predictions (Talvitie, 2014, 2016) or predict many timesteps into the future (Whitney & Fergus, 2018).”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, methodology): “In such domains, the performance of model-based agents employing standard planning methods usually suffers from model errors resulting from function approximation [18, 19].”
Compress and Control (2014, methodology): “The main difficulty that arises when using learnt forward models is that the modeling errors tend to compound when reasoning over long time horizons (Talvitie 2014).”
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, methodology): “…of increasingly effective approaches for learning predictive models in Atari Games (Bellemare, Veness, & Bowling, 2013; Bellemare, Veness, & Talvitie, 2014; Oh et al., 2015), to our knowledge this is the ﬁrst time that a learned dynamics model has been successfully used for planning in…”
Learning Powerful Policies by Using Consistent Dynamics Model (2019, background): “In order to make a prediction about the future, dynamics models are unrolled step by step which leads to “compounding errors” (Talvitie, 2014; Bengio et al., 2015; Lamb et al., 2016): an error in modeling the environment at time t affects the predicted observations at all subsequent time-steps.”
Combating the Compounding-Error Problem with a Multi-step Model (2019, methodology): “, Unfortunately, when the model is wrong this procedure can interfere with successful planning [Talvitie, 2014; Venkatraman et al. , 2015; Asadi et al. , 2018b].”
Planning with Diffusion for Flexible Behavior Synthesis (2022, background): “Because powerful trajectory optimizers exploit learned models, plans generated by this procedure often look more like adversarial examples than optimal trajectories (Talvitie, 2014; Ke et al., 2018).”
Lipschitz Continuity in Model-based Reinforcement Learning (2018, background): “Even small model errors can be magniﬁed by the planning process resulting in poor performance (Talvitie, 2014).”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, background): “This is referred to as the compounding error phenomenon (Talvitie, 2014; Asadi et al., 2018; Weber et al., 2017).”
Data Efficient Reinforcement Learning for Legged Robots (2019, methodology): “Instead, we introduce a multi-step loss function to train the dynamics model, similar to [17, 18].”
An investigation of model-free planning (2019, background): “Model-based approaches are potentially powerful but have been challenging to scale with learned models in complex and high-dimensional environments (Talvitie, 2014; Asadi et al., 2018), though there has been recent progress in that direction (Buesing et al., 2018; Ebert et al., 2018).”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “Longer horizons carry more information, but present a more difﬁcult prediction problem, as errors accumulate rapidly when a model is applied to its own previous outputs (Talvitie, 2014).”
Deep Reinforcement Learning (2018, background): “Variants of scheduled sampling (Bengio et al., 2015) may be used to improve robustness of these models, but scaling to complex domains has proven challenging (Talvitie, 2014).”
Iterative Value-Aware Model Learning (2018, background): “Talvitie [2014], Venkatraman et al. [2015] suggest hallucinated replay to mitigate this issue.”