Action Branching Architectures for Deep Reinforcement Learning

By Arash Tavakoli, Fabio Pardo, Petar Kormushev. 2017

334 citations recorded by the source corpus.

Discrete-action algorithms have been central to numerous recent successes of deep reinforcement learning. However, applying these algorithms to high-dimensional action tasks requires tackling the combinatorial increase of the number of possible actions with the number of action dimensions. This problem is further exacerbated for continuous-action tasks that require fine control of actions via discretization. In this paper, we propose a novel neural architecture featuring a shared decision module followed by several network branches, one for each action dimension. This approach achieves a linear increase of the number of network outputs with the number of degrees of freedom by allowing a level of independence for each individual action dimension. To illustrate the approach, we present a novel agent, called Branching Dueling Q-Network (BDQ), as a branching variant of the Dueling Double Deep Q-Network (Dueling DDQN). We evaluate the performance of our agent on a set of challenging continuous control tasks. The empirical results show that the proposed agent scales gracefully to environments with increasing action dimensionality and indicate the significance of the shared decision module in coordination of the distributed action branches. Furthermore, we show that the proposed agent performs competitively against a state-of-the-art continuous control algorithm, Deep Deterministic Policy Gradient (DDPG).

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “In contrast with our method, work has been done on adapting deep Q-learning methods for continuous actions, which replace selection of the action with the highest estimated value with various approximations (Gu et al., 2016; Metz et al., 2017; Tavakoli et al., 2017).”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “[64] Arash Tavakoli, Fabio Pardo, and Petar Kormushev.”
Continuous Control with Action Quantization from Demonstrations (2021, methodology): “Various methods have addressed this limitation by making the strong assumption of independence (Tavakoli et al., 2018; Tang & Agrawal, 2020; Andrychowicz et al., 2020) or of causal dependence (Metz et al.”
Q-Learning in enormous action spaces via amortized approximate maximization (2020, methodology): “Tavakoli et al. (2017) model Q-values for all sub-actions using a shared state value and a sub-action-speciﬁc advantage parametrization.”
CESMA: Centralized Expert Supervises Multi-Agents (2019, background): “Other techniques such as action branching [47] have also been considered.”
Growing Action Spaces (2019, methodology): “One option would be to use the action-branching architecture of (Tavakoli et al., 2018), which in effect carries out independent Q-learning (Tan,”