Planning with Expectation Models
By Yi Wan, Muhammad Zaheer, Adam White, Martha White, Richard S. Sutton. 2019
26 citations recorded by the source corpus.
Distribution and sample models are two popular model choices in model-based reinforcement learning (MBRL). However, learning these models can be intractable, particularly when the state and action spaces are large. Expectation models, on the other hand, are relatively easier to learn due to their compactness and have also been widely used for deterministic environments. For stochastic environments, it is not obvious how expectation models can be used for planning as they only partially characterize a distribution. In this paper, we propose a sound way of using approximate expectation models for MBRL. In particular, we 1) show that planning with an expectation model is equivalent to planning with a distribution model if the state value function is linear in state features, 2) analyze two common parametrization choices for approximating the expectation: linear and non-linear expectation models, 3) propose a sound model-based policy evaluation algorithm and present its convergence results, and 4) empirically demonstrate the effectiveness of the proposed planning algorithm.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matteo Hessel
- John Aslanides
- Hado van Hasselt
- Pieter Abbeel
- Sergey Levine
- David Silver
- Shimon Whiteson
- Chelsea Finn
In Computational Theory and Mathematics
In Reinforcement Learning
- Matteo Hessel
- John Aslanides
- Hado van Hasselt
- Pieter Abbeel
- Sergey Levine
- David Silver
- Marc G. Bellemare
- R´emi Munos
Citations in the corpus
- When to use parametric models in reinforcement learning? (2019, background): “e steps may be unhelpful, as an expected state may itself not be a valid state; the output of a model may not have useful semantics when using an expected state as input rather than a real state (cf. Wan et al., 2019). Planning is associated with models, because a common way to use computation to im”