Action-Gap Phenomenon in Reinforcement Learning

By Amir-massoud Farahmand. 2011

69 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, background): “In particular, as the action gap of an operator increases while remaining optimality-preserving, the end result is greater robustness to approximation or estimation errors [8].”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “Increasing the action gap is advantageous in the presence of approximation or estimation error (Farahmand 2011), and may be crucial for systems operating at a fine time scale such as video games (Togelius et al.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “…on how to design effective RL exploration policies (Weng, 2020), in this work we used a simple UCB-style exploration criterion (Chen et al., 2017) as follows: explore ( s, a ) = mean Given N sampled actions from the modiﬁed proposal distribution, we take the action with highest Q explore.”
Multi-agent reinforcement learning as a rehearsal for decentralized planning (2016, neutral): “The original proposal of action-gap regularity [11] required a polynomial upper bound on Pð0oGirεÞ in order to establish a contraction operation in value updates for single agent RL.”
Value-Aware Loss Function for Model-based Reinforcement Learning (2017, background): “This is due to the action-gap phenomenon [Farahmand, 2011].”
Munchausen Reinforcement Learning (2020, background): “(6) AL aims at increasing the action-gap [13] defined as the difference, for a given state, between the (optimal) value of the optimal action and that of the suboptimal ones.”
On Inductive Biases in Deep Reinforcement Learning (2019, background): “1) Operating at a slower timescale increases the action gap (Farahmand, 2011), which can lead to more stable learning (Bellemare et al., 2015) because it becomes easier to appropriately rank actions reliably when the value estimates are uncertain or noisy.”
Faster Rates for Policy Learning (2017, background): “Notable exceptions to this n restriction occur for outcome weighted learning (OWL) methods under a hard margin [2] and rates attainable by plug-in estimation strategies [6, 7, 8].”
Regularized Policy Iteration with Nonparametric Function Spaces (2016, methodology): “Greedy algorithms have recently been developed for the value function estimation by Johns (2010); Painter-Wakefield and Parr (2012); Farahmand and Precup (2012); Geramifard et al.”
Regularization in reinforcement learning (2011, background): “2 in which the material is mostly borrowed from Chapter 4 (also Farahmand and Szepesvári [2012]).”