Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning

By Aravind S. Lakshminarayanan, Balaraman Ravindran. 2017

101 citations recorded by the source corpus.

Reinforcement Learning algorithms can learn complex behavioral patterns for sequential decision making tasks wherein an agent interacts with an environment and acquires feedback in the form of rewards sampled from it. Traditionally, such algorithms make decisions, i.e., select actions to execute, at every single time step of the agent-environment interactions. In this paper, we propose a novel framework, Fine Grained Action Repetition (FiGAR), which enables the agent to decide the action as well as the time scale of repeating it. FiGAR can be used for improving any Deep Reinforcement Learning algorithm which maintains an explicit policy estimate by enabling temporal abstractions in the action space. We empirically demonstrate the efficacy of our framework by showing performance improvements on top of three policy search algorithms in different domains: Asynchronous Advantage Actor Critic in the Atari 2600 domain, Trust Region Policy Optimization in Mujoco domain and Deep Deterministic Policy Gradients in the TORCS car racing domain.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “Fine-grained action repetition (Sharma, Lakshminarayanan, and Ravindran 2017) enabled to learn how to repeat actions.”
Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space (2018, background): “Sharma, Lakshminarayanan, and Ravindran did a similar approach where they extended TRPO to control the repetition of the action, that is, how many steps an action should execute (2017).”
Multi-Level Discovery of Deep Options (2017, background): “Empirically, we ﬁnd that this assumption does not sacriﬁce the quality of the discovered options (Section 5.1).”
Mastering diverse control tasks through world models (2025, unknown): “Although this challenge could be overcome with specific inductive biases, such as learned action repeat 65 , we argue that learning to keep the same button pressed for hundreds of steps does not lie at the core of what makes Minecraft an interesting challenge for artificial intelligence.”
Online Multi-Task Learning Using Active Sampling (2017, background): “A more complete version of this manuscript can be found at (Sharma & Ravindran, 2017)”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, methodology): “We use action repetition [Sharma et al., 2017] to introduce irregular transition times to Mujoco tasks, however, we assume the intermediate observations are not available so that the dynamics naturally ﬁts into the SMDP deﬁnition.”
Learning to Multi-Task by Active Sampling (2017, methodology): “All the target scores in this work were taken from Table 4 of [Sharma et al., 2017].”
Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning (2017, methodology): “Sharma et al. [2017] have demonstrated remarkable improvements in a variety of domains using a framework that factors the policy into one for choosing actions and another for choosing repetitions.”