Time Limits in Reinforcement Learning

By Fabio Pardo, Arash Tavakoli, Vitaly Levdik, Petar Kormushev. 2017

196 citations recorded by the source corpus.

In reinforcement learning, it is common to let an agent interact for a fixed amount of time with its environment before resetting it and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed period, or (ii) an indefinite period where time limits are only used during training to diversify experience. In this paper, we provide a formal account for how time limits could effectively be handled in each of the two cases and explain why not doing so can cause state-aliasing and invalidation of experience replay, leading to suboptimal policies and training instability. In case (i), we argue that the terminations due to time limits are in fact part of the environment, and thus a notion of the remaining time should be included as part of the agent's input to avoid violation of the Markov property. In case (ii), the time limits are not part of the environment and are only used to facilitate learning. We argue that this insight should be incorporated by bootstrapping from the value of the state at the end of each partial episode. For both cases, we illustrate empirically the significance of our considerations in improving the performance and stability of existing reinforcement learning algorithms, showing state-of-the-art results on several control tasks.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

Citations in the corpus

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, background): “4 where one assumes for the algorithm an infinite time horizon but then trains using a finite time horizon [16].”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, neutral): “We also terminate if the agent has failed to reach the goal after 20 time steps, but treat the two types of termination differently when computing the TD error (see Pardo et al. (2017)).”
Automated Machine Learning: Methods, Systems, Challenges (2019, unknown): “We have also rescaled the time step index into a real number from [−1, 1] and included it into observation vector as was recently proposed by [33].”
Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (2021, methodology): “This solution has been discussed in [24], but interestingly, this distinction is not part of the widely used Gym environment interface [25] and is ignored by popular implementations such as Stable-Baselines [26]1.”
Learning Improvement Heuristics for Solving Routing Problems (2019, background): “Additionally, since we do not deﬁne terminal state, it is necessary to bootstrap the value from the time limit state, such that the policy for the continuing task can be learned correctly [35].”
What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (2021, methodology): “B.4 where one assumes for the algorithm an inﬁnite time horizon but then trains using a ﬁnite time horizon [16].”