Discrete Sequential Prediction of Continuous Actions for Deep RL

By Luke Metz, Julian Ibarz, Navdeep Jaitly, James Davidson. 2017

135 citations recorded by the source corpus.

It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequence-to-sequence models for structured prediction problems to develop policies over discretized spaces. Central to this method is the realization that complex functions over high dimensional spaces can be modeled by neural networks that predict one dimension at a time. Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions. With this parameterization, it is possible to both leverage the compositional structure of action spaces during learning, as well as compute maxima over action spaces (approximately). On a simple example task we demonstrate empirically that our method can perform global search, which effectively gets around the local optimization issues that plague DDPG. We apply the technique to off-policy (Q-learning) methods and show that our method can achieve the state-of-the-art for off-policy methods on several continuous control tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “In contrast with our method, work has been done on adapting deep Q-learning methods for continuous actions, which replace selection of the action with the highest estimated value with various approximations (Gu et al., 2016; Metz et al., 2017; Tavakoli et al., 2017).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Our best results on Trust-PCL are able to maintain the stability and solution quality of TRPO while approaching the sample-efficiency of value-based methods (see e.g., Metz et al. (2017)).”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, methodology): “Our methods can also be used directly with any other Q-learning methods ((Watkins & Dayan, 1992; Mnih et al., 2013; Gu et al., 2017; Amos et al., 2016; Metz et al., 2017)).”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “Inspired by recent works (Metz et al., 2017; Van de Wiele et al., 2020), our generative model architecture also makes use of discretization in each action dimension.”
Hindsight Experience Replay (2017, methodology): “Algorithm 1 Hindsight Experience Replay (HER) Given: • an off-policy RL algorithm A , e.g. DQN, DDPG, NAF, SDQN • a strategy S for sampling goals for replay, e.g.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “[81] used this idea in order to construct the sequential DQN,”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, methodology): “The TDM can be trained with any off-policy Q-learning algorithm, such as DQN (Mnih et al., 2015), DDPG (Lillicrap et al., 2015), NAF (Gu et al., 2016), and SDQN (Metz et al., 2017).”
Discrete Autoencoders for Sequence Models (2018, background): “Even in reinforcement learning where action spaces are naturally continuous, Metz et al. (2017) show that discretizing them and using autoregressive models can yield improvements.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, neutral): “Previous works attempted to leverage this observation by using auto-regressive models or assuming conditional independence between the groups [34, 58].”
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control (2017, methodology): “To our knowledge, there are few works [9] which reproduce existing policy gradients methods in reinforcement learning, yet many use as these algorithms as baselines to compare their novel work against [9, 10, 11, 12].”
Grandmaster level in StarCraft II using multi-agent reinforcement learning (2019, methodology): “To manage the structured, combinatorial action space, the agent uses an auto-regressive policy 7,10,11 and recurrent pointer network 12 .”
Continuous Control with Action Quantization from Demonstrations (2021, background): “…is to assume that action dimensions are independent (Tavakoli et al., 2018; Andrychowicz et al., 2020; Vieillard et al., 2021; Tang & Agrawal, 2020), or to assume or learn a causal dependence between them (Metz et al., 2017; Tessler et al., 2019; Sakryukin et al., 2020; Tavakoli et al., 2021).”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration (2023, methodology): “Prior work has addressed this by exploring different action representations, such as autoregressive predictions of quantized actions [24–27], using latent variable models [17, 21], switching to an implicit policy representation [28], and, most recently, using conditional diffusion models for…”
NerveNet: Learning Structured Policy with Graph Neural Networks (2018, background): “1 INTRODUCTION Deep reinforcement learning (RL) has received increasing attention over the past few years, with the recent success of applications such as playing Atari Games, Mnih et al. (2015), and Go, (Silver et al.”
Q-Learning in enormous action spaces via amortized approximate maximization (2020, methodology): “Notably, Metz et al. (2018) conditions the lower-level Q function on sub-actions already taken as part of the current macro-action; we experiment with a similar strategy for our learned proposal distribution.”
Smoothed Action Value Functions for Learning Gaussian Policies (2018, unknown): “…functions to stably learn a policy, which in the past has been used to either approximate expected (Rummery & Niranjan, 1994; Van Seijen et al., 2009; Gu et al., 2017) or optimal (Watkins, 1989; Silver et al., 2014; Nachum et al., 2017; Haarnoja et al., 2017; Metz et al., 2017) future value.”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, background): “However modeling such a combinatorial action space in the context of sequential recommendations poses challenges to existing RL algorithms [Sunehag et al., 2015, Metz et al., 2017], as the assumptions they make render them ineffective for exploration and generalization in large-scale recommenders.”