Deep Reinforcement Learning in Parameterized Action Space

By Matthew J. Hausknecht, Peter Stone. 2015

343 citations recorded by the source corpus.

Recent work has shown that deep neural networks are capable of approximating both value functions and policies in reinforcement learning domains featuring continuous state and action spaces. However, to the best of our knowledge no previous work has succeeded at using deep neural networks in structured (parameterized) continuous action spaces. To fill this gap, this paper focuses on learning within the domain of simulated RoboCup soccer, which features a small set of discrete action types, each of which is parameterized with continuous variables. The best learned agent can score goals more reliably than the 2012 RoboCup champion agent. As such, this paper represents a successful extension of deep reinforcement learning to the class of parameterized action space MDPs.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces (2019, methodology): “Previous work [ Hausknecht and Stone, 2016; Wang et al. , 2018; Wei et al. , 2018b] applied RL to the single-agent version of HFO, letting one agent try to goal without a goal-keeper (1v0).”
Guided Deep Reinforcement Learning for Swarm Systems (2017, methodology): “Furthermore, we use the technique of inverting gradients introduced by [2] to bound the parameters of the action space.”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “In MARL, several agents interact and learn in an environment simultaneously, either competitively such as in Go [91] and Poker [39, 105, 74], cooperatively such as when learning to communicate [23, 93, 36], or some mix of the two [60, 95, 35].”
Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, background): “…68 ), or non-learning systems ( 50, 69 ), and previously studied multi-agent domains often consist of discrete-state environments ( 18, 38, 64 ), have simpliﬁed 2D dynamics ( 23, 41, 53 ) or have fully observable or non-perceptual features ( 18, 23, 41, 48, 53, 64 ) rather than pixel observations.”
Exponentially Weighted Imitation Learning for Batched Historical Data (2018, methodology): “Like in [Hausknecht and Stone, 2016], we let the agent try to goal without a goalkeeper.”
Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space (2018, methodology): “• Note that P-DQN can use human players’ data, while it is hard to use human players’ data in Hausknecht and Stone [2016] because there is only discrete action k without parameters f .”
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, unknown): “…real-world applications involve a moderately large population of agents, such as algorithmic trading (Wellman et al., 2005), sport team competition (Hausknecht & Stone, 2015), and humanitarian assistance and disaster response (Meier, 2015), where one agent should collaborate and/or compete with…”
Terrain-adaptive locomotion skills using deep reinforcement learning (2016, methodology): “Another recent work proposes the use of policy-gradient DNN-RL with parameterized controllers for simulated robot soccer [Hausknecht and Stone 2015], and theoretically-grounded algorithms have been proposed to ensure monotonic policy improvement [Schulman et al.”
Continuous Deep Q-Learning with Model-based Acceleration (2016, background): “…of challenging problems (Kober & Peters, 2012; Deisenroth et al., 2013), and has recently been extended to handle large neural network policies and value functions (Mnih et al., 2015; Lillicrap et al., 2016; Wang et al., 2015; Heess et al., 2015; Hausknecht & Stone, 2015; Schulman et al., 2015).”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, methodology): “One approach for constructing a hierarchical policy is by using a mixture-of-experts model [15, 19, 28, 31, 42], where the composite policy’s action distribution π(a|s, g) is represented by a weighted sum of distributions from a set of primitives πi(a|s, g) (i.”
Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space (2018, methodology): “We implemented the Parameterization Action DDPG † (PADDPG) algorithm from (Hausknecht and Stone 2015) as our comparison baseline which is considered as the state of the art.”