Variable Resolution Discretization in Optimal Control

By R´emi Munos, Andrew W. Moore. 2002

376 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “…underpins Baird’s advantage learning (Baird 1999), designed for continuous time control, and occurs naturally when considering the discretized solution of continuous time and space MDPs (e.g. Munos and Moore 1998; 2002), whose limit is the Hamilton-Jacobi-Bellman equation (Kushner and Dupuis 2001).”
Dynamic Programming for Structured Continuous Markov Decision Problems (2004, background): “The most common approaches to continuous state variables in MDPs are to use function approximators (Gordon 1995; Boyan & Moore 1995), such as artificial neural networks, to discretize the continuous state space more or less naively, which does not scale well to multiple dimensions, or to use Monte-C”
An Introduction to Deep Reinforcement Learning (2018, positive): “Previous RL approaches had a difficult design issue in the choice of features (Munos and Moore, 2002; Bellemare et al., 2013).”
Using dynamic programming with adaptive grid scheme for optimal control problems in economics (2004, methodology): “Perhaps closest to our approach are the techniques discussed in Munos and Moore (2002).”
Policy compression for aircraft collision avoidance systems (2016, methodology): “Discretized score tables like this have been compressed with Gaussian processes [6] and decision trees [7].”
Adaptive Tile Coding for Value Function Approximation (2007, methodology): “…representations and refines them over time, a strategy that has proven effective for other function approximators, such as neural networks (Whiteson & Stone 2006), piecewise-linear representations based on kd-trees (Munos & Moore 2002), and uniform grid discretizations (Chow & Tsitsiklis 1991).”
Closed-Loop Learning of Visual Control Policies (2011, neutral): “Variable Resolution Grids are probably the most advanced adaptive-resolution algorithm available so far.”
Automated Planning and Acting (2016, methodology): “The latter have been addressed with severable discretization techniques such as adaptive approximation by Munos and Moore [437], piecewise constant or linear approximation by Feng et al.”
Growing Action Spaces (2019, methodology): “nvironment. In reinforcement learning, this has often taken the form of adaptively growing the resolution of the state space considered by a piecewise constant discretised approximation [Moore, 1994, Munos and Moore, 2002, Whiteson et al., 2007]. Stanley and Miikkulainen [2004] study continual compl”
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions (2020, neutral): “In the context of optimal control and RL, inﬂuence functions were ﬁrst introduced by Munos & Moore (2002) to aid in online optimization of policies.”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, background): “So the question of choosing the right BFs is still a relevant one, see e.g. Munos and Moore (2002); Gr¨une (2004); Bertsekas and Yu (2009).”
Variance Reduction Methods for Sublinear Reinforcement Learning (2018, methodology): “We will be applying the law of total variance (see e.g., [MM02, AOM17]) to bound The rest of proof follows from choosing β = 4.”
Non-Parametric Approximate Linear Programming for MDPs (2011, methodology): “Other non-parametric approaches to reinforcement learning include variable resolution grids (Munos and Moore 2002), the work of Ormoneit and Sen (2002), Fitted Q- Iteration with tree-based approximators (Ernst, Geurts, and Wehenkel 2005) and many kernelized methods (Taylor and Parr 2009).”
Planning algorithms (2006, background): “Related work exists in artificial intelligence [725] and control theory [946].”
Lyapunov Design for Safe Reinforcement Learning (2003, background): “Using a variety of tricks, including intelligentlyallocated variable-resolution grids, Munos and Moore (2002) recently reported being able to solve problems with at most four- or, to a degree, five-dimensional state spaces.”
Reinforcement Learning on Slow Features of High-Dimensional Input Streams (2010, background): “There, the main approach is either to reduce the size of a discrete state space or to discretize a continuous state-space [59,60].”
Automatic basis function construction for approximate dynamic programming and reinforcement learning (2006, background): “Moreover, even with careful engineering, continuous problems with more than 10-12 dimensions cannot be handled by current techniques (Munos & Moore, 2002).”