Iterative Hierarchical Optimization for Misspecified Problems (IHOMP)

By Daniel Jaymin Mankowitz, Timothy A. Mann, Shie Mannor. 2016

10 citations recorded by the source corpus.

For complex, high-dimensional Markov Decision Processes (MDPs), it may be necessary to represent the policy with function approximation. A problem is misspecified whenever, the representation cannot express any policy with acceptable performance. We introduce IHOMP : an approach for solving misspecified problems. IHOMP iteratively learns a set of context specialized options and combines these options to solve an otherwise misspecified problem. Our main contribution is proving that IHOMP enjoys theoretical convergence guarantees. In addition, we extend IHOMP to exploit Option Interruption (OI) enabling it to decide where the learned options can be reused. Our experiments demonstrate that IHOMP can find near-optimal solutions to otherwise misspecified problems and that OI can further improve the solutions.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Industrial and Manufacturing Engineering

Citations in the corpus

A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “Recent work in RL has provided insights into learning reusable skills [Mankowitz et al., 2016a; Mankowitz et al., 2016b], but this has been limited to low dimensional problems.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “This return-equivalent MDP is easier to learn, is guaranteed to have the same optimal policy as the original MDP, and the approach shows improvements in Atari tasks with long delays.”
Representation Learning for Grounded Spatial Reasoning (2017, methodology): “…T (s|st−1,a)V̂ (s,x;Θ) 9: Observe next state st and reward rt 10: Store trajectory (s=s0,s1,...,r=r0,r1,...) in D 11: for j=1,J do 12: Sample random trajectory (s,r) from D 13: Perform gradient descent step on loss L(θ) world reinforcement learning benchmark (Sutton, 1996; Mankowitz et al., 2016).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “Previous research in this area includes learning the options from scratch [Mankowitz et al. 2016a;b; Bacon et al. 2017] as well as planning, given a pre-trained set of options [Schaul et al. 2015; Mankowitz et al. 2018b].”