Q-Learning in enormous action spaces via amortized approximate maximization
By Tom Van de Wiele, David Warde-Farley, Andriy Mnih, Volodymyr Mnih. 2020
67 citations recorded by the source corpus.
Applying Q-learning to high-dimensional or continuous action spaces can be difficult due to the required maximization over the set of possible actions. Motivated by techniques from amortized inference, we replace the expensive maximization over all actions with a maximization over a small subset of possible actions sampled from a learned proposal distribution. The resulting approach, which we dub Amortized Q-learning (AQL), is able to handle discrete, continuous, or hybrid action spaces while maintaining the benefits of Q-learning. Our experiments on continuous control tasks with up to 21 dimensional actions show that AQL outperforms D3PG (Barth-Maron et al, 2018) and QT-Opt (Kalashnikov et al, 2018). Experiments on structured discrete action spaces demonstrate that AQL can efficiently learn good policies in spaces with thousands of discrete actions.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shimon Whiteson
- Sergey Levine
- Philip H. S. Torr
- Christian Schröder de Witt
- Gregory Farquhar
- Wendelin Böhmer
- Bei Peng
- Gabriel Synnaeve
Citations in the corpus
- EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “The architecture for our autoregressive generative model is inspired by the works of (Metz et al., 2017; Van de Wiele et al., 2020; Germain et al., 2015).”
- Growing Action Spaces (2019, background): “A number of other works address the problem of generalisation and representation for value functions with large discrete action spaces, without explicitly addressing the resulting exploration problem (Dulac-Arnold et al., 2015; Pan et al., 2018; Van de Wiele et al., 2020).”
- Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, methodology): “Future work will investigate the utility of recent amorti-sation techniques for cross-entropy maximisation (Van de Wiele et al., 2020) and the relationship between exploration strategies in continuous Q -learning and deterministic policy gradient approaches.”