Combating the Compounding-Error Problem with a Multi-step Model
By Kavosh Asadi, Dipendra Kumar Misra, Seungchan Kim, Michael L. Littman. 2019
71 citations recorded by the source corpus.
Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.
Subfields: Artificial Intelligence, Software, Statistics and Probability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Igor Mordatch
- Quoc V. Le
- Josh Tenenbaum
In Software
In Statistics and Probability
- Theophane Weber
- Karol Gregor
- Nan Rosemary Ke
- Pieter Abbeel
- Sergey Levine
- Danilo Jimenez Rezende
- Lars Buesing
- Igor Mordatch
Citations in the corpus
- Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “This family of models, sometimes called models with overshoot , can for instance be found in (Oh et al., 2017; Luo et al., 2019; Guo et al., 2018; Hafner et al., 2018; Gregor et al., 2019; Asadi et al., 2019) and MuZero (Schrittwieser et al., 2019) and is illustrated in Figure 3(d).”
- Mastering Atari Games with Limited Data (2021, background): “[2] proposed a multi-step model to combat the compounding error.”
- γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, background): “There is a natural tradeoff with γ -models: the higher γ is, the fewer model steps are needed to make long-horizon predictions, reducing model-based compounding prediction errors (Asadi et al., 2019; Janner et al., 2019).”