The utility of temporal abstraction in reinforcement learning
By Nicholas K. Jong, Todd Hester, Peter Stone. 2008
89 citations recorded by the source corpus.
The hierarchical structure of real-world problems has motivated extensive research into temporal abstractions for reinforcement learning, but precisely how these abstractions allow agents to improve their learning performance is not well understood. This paper investigates the connection between temporal abstraction and an agent's exploration policy, which determines how the agent's performance improves over time. Experimental results with standard methods for incorporating temporal abstractions show that these methods benefit learning only in limited contexts. The primary contribution of this paper is a clearer understanding of how hierarchical decompositions interact with reinforcement learning algorithms, with important consequences for the manual design or automatic discovery of action hierarchies.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Sergey Levine
- Ofir Nachum
- Pieter Abbeel
- Yoshua Bengio
- Shimon Whiteson
- Chelsea Finn
- J. Urgen Schmidhuber
- Shixiang Shane Gu
Citations in the corpus
- Value Preserving State-Action Abstractions (2020, background): “However, by including options and primitive actions, learning algorithms face a larger branching factor, and must search the full space of policies which can hurt learning performance (Jong et al., 2008).”
- Context-Aware Policy Reuse (2018, background): “od, it cannot learn when to terminate the sub-policies autonomously. Moreover, MLSH has no theoretical guarantee for optimal convergence. Option Learning. In contrast to the works of option discovery [21, 29, 30, 31], CAPS focuses on multi-policy reuse. Although some methods have been proposed to re”
- Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? (2019, methodology): “Following a previous flurry of research, empirical demonstrations, and claimed intuitive benefits of options in the early 2000’s, Jong et al. (2008) set out to systematically evaluate these techniques.”
- Multitask Soft Option Learning (2019, result): “The finding that access to misspecified, hard options can actually hurt exploration is consistent with previous literature (Jong et al., 2008).”