Deep Abstract Q-Networks
By Melrose Roderick, Christopher Grimm, Stefanie Tellex. 2017
35 citations recorded by the source corpus.
We examine the problem of learning and planning on high-dimensional domains with long horizons and sparse rewards. Recent approaches have shown great successes in many Atari 2600 domains. However, domains with long horizons and sparse rewards, such as Montezuma's Revenge and Venture, remain challenging for existing methods. Methods using abstraction (Dietterich 2000; Sutton, Precup, and Singh 1999) have shown to be useful in tackling long-horizon problems. We combine recent techniques of deep reinforcement learning with existing model-based approaches using an expert-provided state abstraction. We construct toy domains that elucidate the problem of long horizons, sparse rewards and high-dimensional inputs, and show that our algorithm significantly outperforms previous methods on these domains. Our abstraction-based approach outperforms Deep Q-Networks (Mnih et al. 2015) on Montezuma's Revenge and Venture, and exhibits backtracking behavior that is absent from previous methods.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Percy Liang
- Sergey Levine
- Panupong Pasupat
- Yoshua Bengio
- J. Urgen Schmidhuber
- Shimon Whiteson
- Quoc V. Le
- Kelvin Guu
Citations in the corpus
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “They can also be readily extended to be compatible with hierarchical RL (Roderick et al., 2017; Vezhnevets et al., 2017).”
- Strategic Object Oriented Reinforcement Learning (2018, methodology): “Model free methods to use object representation [9, 23, 5] fail to scale to large MDPs and do not leverage object”
- Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration (2018, background): “Main HRL frameworks include abstract actions (Sutton et al., 1999; Konidaris & Barto, 2007; Hauser et al., 2008), abstract partial policies (Parr & Russell, 1998), and abstract states (Roderick et al., 2017; Dietterich, 1998; Li et al., 2006).”
- Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning (2019, background): “Furthermore, Pathak et al. (2017) use the predictability of the observed transition as intrinsic reward signal, and Roderick et al. (2018) reduce uncertainty with prior knowledge over state abstractions.”