Self-Correcting Models for Model-Based Reinforcement Learning

By Erik Talvitie. 2016

104 citations recorded by the source corpus.

When an agent cannot represent a perfectly accurate model of its environment's dynamics, model-based reinforcement learning (MBRL) can fail catastrophically. Planning involves composing the predictions of the model; when flawed predictions are composed, even minor errors can compound and render the model useless for planning. Hallucinated Replay (Talvitie 2014) trains the model to "correct" itself when it produces errors, substantially improving MBRL with flawed models. This paper theoretically analyzes this approach, illuminates settings in which it is likely to be effective or ineffective, and presents a novel error bound, showing that a model's ability to self-correct is more tightly related to MBRL performance than one-step prediction error. These results inspire an MBRL algorithm for deterministic MDPs with performance guarantees that are robust to model class limitations.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “Unfortunately, to date, it has proven difﬁcult to learn models for such domains with sufﬁcient ﬁdelity to realise the beneﬁts of look-ahead planning (Oh et al., 2015; Talvitie, 2017).”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Training set augmentation (Talvitie, 2014, 2017; Venkatraman, Hebert, & Bagnell, 2015) has shown that it is possible to improve an otherwise limited model.”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “Alternatively, a model may also be trained on its own outputs to avoid compounding error from multi-step predictions (Talvitie, 2014, 2016) or predict many timesteps into the future (Whitney & Fergus, 2018).”
Learning One Representation to Optimize All Rewards (2021, background): “Learning a model of the world is another possibility, but it still requires explicit planning for each new reward; moreover, synthesizing accurate trajectories of states over long time ranges has proven difﬁcult [Tal17, KST + 18].”
Lipschitz Continuity in Model-based Reinforcement Learning (2018, background): “Previous work has shown that model error can result in poor long-horizon predictions and ineffective planning (Talvitie, 2014; 2017).”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “Planning with inaccurate models is detrimental as the model errors can compound due to rollouts, resulting in misleading agent updates (Talvitie, 2017; Jafferjee et al., 2020; Wang et al., 2021; Liu et al., 2023).”
Combating the Compounding-Error Problem with a Multi-step Model (2019, background): “Speciﬁcally, we show a bound on value-function estimation error of a ﬁxed policy in terms of the error in the agent’s model, while highlighting similar bounds from prior work [Ross et al. , 2011; Talvitie, 2017; Asadi et al. , 2018b].”
Deep Reinforcement Learning (2018, background): “Unfortunately, to date, it has proven difficult to learn models for such domains with sufficient fidelity to realise the benefits of look-ahead planning (Oh et al., 2015; Talvitie, 2017).”
Model Primitive Hierarchical Lifelong Reinforcement Learning (2019, background): “The world however is complex, and learning models that are consistent enough to plan with is not only hard [24], but planning with such one-step models is suboptimal [11].”
Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains (2018, methodology): “…approaches such as PILCO [Deisenroth and Rasmussen, 2011], Hallucinated-Dagger [Talvitie, 2017], and CPSRs [Hamilton et al., 2014], unlike ER, can rollout unlikely trajectories ending up in hypothetical states that do not match any real state in the world when the model is wrong [Talvitie, 2017].”