When to use parametric models in reinforcement learning?

By Hado van Hasselt, Matteo Hessel, John Aslanides. 2019

211 citations recorded by the source corpus.

We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan: to use more computation without additional data to improve predictions and behaviour. We discuss when to expect benefits from either approach, and interpret prior work in this context. We hypothesise that, under suitable conditions, replay-based algorithms should be competitive to or better than model-based algorithms if the model is used only to generate fictional transitions from observed states for an update rule that is otherwise model-free. We validated this hypothesis on Atari 2600 video games. The replay-based algorithm attained state-of-the-art data efficiency, improving over prior results with parametric models.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “None of these prior methods has constructed a model that facilitates effective planning in visually complex domains such as Atari; results lag behind well-tuned, model-free methods, even in terms of data efficiency [45].”
Model-Based Reinforcement Learning for Atari (2019, background): “The results are on a par with SimPLe – both of the model-free methods are better in 13 games, while SimPLe is better in the other 13 out of the total 26 games tested (note that in Section 4.2 van Hasselt et al. (2019) compares with the results of our ﬁrst preprint, later improved).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “This can be thought of as a scaled-down and simplified version of the AlphaZero algorithm (Silver et al., 2018), optionally coupled with an environment transition model, similar to the set-up in (van Hasselt et al., 2019).”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “In the Atari setting benchmarked at 100k interaction steps, we show that CURL coupled with a data-efficient version of Rainbow DQN (van Hasselt et al., 2019) results in 1.2x median higher performance over prior methods such as SimPLe (Kaiser et al., 2019), improving upon Efficient Rainbow (van…”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, background): “Therefore, model-based reinforcement learning (MBRL) (Janner et al., 2019; Kaiser et al., 2019; Schrittwieser et al., 2020; Zhang et al., 2019b; van Hasselt et al., 2019; Hafner et al., 2019b;a; Lenz et al., 2015), which explicitly builds a predictive model to generate samples for learning RL…”
Revisiting Fundamentals of Experience Replay (2020, background): “…(2019) report that agent performance is sensitive to the number of environment steps taken per gradient step, with too small or large a ratio also hindering performance. van Hasselt et al. (2019) vary this ratio in combination with batch sizes to obtain a more sample-efficient version of Rainbow.”
An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “Experience replay-based algorithms can be more sample efficient than model-based approaches (Van Hasselt et al., 2019), and using the DQN Replay Dataset on Atari 2600 games for designing non-parametric replay models (Pan et al., 2018) and parametric world models (Kaiser et al., 2019) is another…”
Model-augmented Prioritized Experience Replay (2022, methodology): “Moreover, since Rainbow (van Hasselt et al., 2019) uses the multi-step return with length 20 to compute the TD-error instead of the reward, we also use it instead of the estimated reward.”
Mastering Atari Games with Limited Data (2021, background): “Many previous works even suspect whether model-based algorithms can really offer data efficiency when using image observations [39].”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, background): “Although in discrete action domains it has been found that ﬁne-tuning DQN variants, including the UTD, can boost performance, little experimental or theoretical analysis is given to explain how this improvement is obtained (Kielak, 2020; van Hasselt et al., 2019).”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “Unfortunately, in stochastic domains, this comes with the challenging task of density estimation over the ground state space, and as a result, performance is about on-par with end-to-end deep RL (Van Hasselt et al., 2019).”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “This observation highlights the impact of model error when the model-generated transitions are used to update the critic, and supports van Hasselt et al. (2019) argument that inaccurate parametric forward dynamics models may be particularly detrimental to value learning.”
Data-Efficient Reinforcement Learning with Self-Predictive Representations (2020, methodology): “Rainbow (Hessel et al., 2018) consolidates these improvements into a single combined algorithm and has been adapted to work well in data-limited regimes (van Hasselt et al., 2019).”