Dynamic Action Repetition for Deep Reinforcement Learning

By A. Srinivas, Balaraman Ravindran. 2017

80 citations recorded by the source corpus.

One of the long standing goals of Artificial Intelligence (AI) is to build cognitive agents which can perform complex tasks from raw sensory inputs without explicit supervision. Recent progress in combining Reinforcement Learning objective functions and Deep Learning architectures has achieved promising results for such tasks. An important aspect of such sequential decision making problems, which has largely been neglected, is for the agent to decide on the duration of time for which to commit to actions. Such action repetition is important for computational efficiency, which is necessary for the agent to respond in real-time to events (in applications such as self-driving cars). Action Repetition arises naturally in real life as well as simulated environments. The time scale of executing an action enables an agent (both humans and AI) to decide the granularity of control during task execution. Current state of the art Deep Reinforcement Learning models, whether they are off-policy or on-policy, consist of a framework with a static action repetition paradigm, wherein the action decided by the agent is repeated for a fixed number of time steps regardless of the contextual state while executing the task. In this paper, we propose a new framework - Dynamic Action Repetition which changes Action Repetition Rate (the time scale of repeating an action) from a hyper-parameter of an algorithm to a dynamically learnable quantity. At every decision-making step, our models allow the agent to commit to an action and the time scale of executing the action. We show empirically that such a dynamic time scale mechanism improves the performance on relatively harder games in the Atari 2600 domain, independent of the underlying Deep Reinforcement Learning algorithm used.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Value Prediction Network (2017, background): “Though using a larger frame-skip generally makes training easier, it may make training harder in some games if they require more fine-grained control [17].”
On Inductive Biases in Deep Reinforcement Learning (2019, background): “A simpler, though more limited, approach is for the agent to learn how long to commit to actions (Lakshminarayanan et al., 2017).”
Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning (2017, unknown): “If one were to extend the action space in a naive way by coupling the actions and the action repetitions, one would end up suffering the kind of action-space blowup as seen in [Lakshminarayanan et al. (2017); Vezhnevets et al.”