Reinforcement Learning with Parameterized Actions

By Warwick Masson, Pravesh Ranchod, George Dimitri Konidaris. 2015

263 citations recorded by the source corpus.

We introduce a model-free algorithm for learning in Markov decision processes with parameterized actions—discrete actions with continuous parameters. At each step the agent must select both which action to use and which parameters to use with that action. We introduce the Q-PAMDP algorithm for learning in these domains, show that it converges to a local optimum, and compare it to direct policy search in the goal-scoring and Platform domains.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Abhinav Kumar Gupta
3. Matthew J. Hausknecht
4. Saurabh Gupta
5. Shimon Whiteson
6. Pieter Abbeel
7. Ruslan Salakhutdinov
8. Peter Stone

In Management Science and Operations Research

1. Ermo Wei
2. Sean Luke
3. Drew Wicke
4. Han Liu
5. Peng Sun
6. Lei Han
7. Qing Wang
8. Jiechao Xiong

Citations in the corpus

• Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces (2019, background): “However, very often the action space in real world is discrete-continuous hybrid, such as Robot soccer [Hausknecht, 2016; Masson et al. , 2016] and Real Time Strategic (RTS) games [Xiong et al. , 2018].”
• Deep Reinforcement Learning in Parameterized Action Space (2015, background): “Masson & Konidaris (2015) present a parameterized-action MDP formulation and approaches for model-free reinforcement learning in such environments.”
• Half Field Offense: An Environment for Multiagent Learning and Ad Hoc Teamwork (2016, background): “Formally characterized as a Parameterized Action Markov Decision Process (PAMDP) [17], HFO’s low-level, parameterized action space features four mutually-exclusive action primitives: Dash, Turn, Tackle, and Kick.”
• Efficient Bimanual Manipulation Using Learned Task Schemas (2019, methodology): “Each task we consider is defined as a parameterized action Markov decision process ( PAMDP ) [10], [26] with finite horizon T .”
• Hierarchy Through Composition with Multitask LMDPs (2017, background): “This scheme may also be interpreted as implementing a parametrized option, but with performance guarantees for any parameter setting (Masson et al., 2016).”
• Cooperation and communication in multiagent deep reinforcement learning (2016, methodology): “Following notation in (Masson and Konidaris, 2015), a Parameterized Action Space Markov Decision Process (PAMDP) is defined by a set of discrete actions Ad = {a1, a2, .”
• Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space (2018, methodology): “Specifically, we consider tasks with parameterized action spaces (Masson, Ranchod, and Konidaris 2016), where at each step the agent must choose both a discrete action and a set of continuous parameters for that action.”
• Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space (2018, methodology): “• Masson et al. [2016] and Khamassi et al. [2017] use on-policy update algorithm for continuous parameters.”