Depth and nonlinearity induce implicit exploration for RL
By Justas Dauparas, Ryota Tomioka, Katja Hofmann. 2018
4 citations recorded by the source corpus.
The question of how to explore, i.e., take actions with uncertain outcomes to learn about possible future rewards, is a key question in reinforcement learning (RL). Here, we show a surprising result: We show that Q-learning with nonlinear Q-function and no explicit exploration (i.e., a purely greedy policy) can learn several standard benchmark tasks, including mountain car, equally well as, or better than, the most commonly-used $\epsilon$-greedy exploration. We carefully examine this result and show that both the depth of the Q-network and the type of nonlinearity are important to induce such deterministic exploration.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Doina Precup
- Scott Fujimoto
- Ofir Nachum
- Shixiang Shane Gu
- Pieter Abbeel
- David Meger
- Aviral Kumar
In Cognitive Neuroscience
In Management Science and Operations Research
- Scott Fujimoto
- Sergey Levine
- Doina Precup
- David Meger
- Ofir Nachum
- Aviral Kumar
- George Tucker
- Shixiang Shane Gu
Citations in the corpus
- Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “It has been previously noted that non-linear function approximation itself induces an implicit exploration without an explicit exploration policy (Dauparas et al., 2018).”