Speedy Q-Learning

By Mohammad Gheshlaghi Azar, R´emi Munos, Mohammad Ghavamzadeh, Hilbert J. Kappen. 2011

212 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “The examples in Section 6 demonstrate the signiﬁcant improvement thus obtained.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “We believe this relationship is the natural extension of the close relationship between tabular value estimation using a decaying step-size (e.g. Even-Dar and Mansour, 2001; Azar et al., 2011) and count-based exploration in the tabular setting (e.”
A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, methodology): “The research literature contains a wide variety of studies of operator alternatives to the Bellman operator, including the -greedy method [22], speedy Q-learning [3], policy iteration-like Q-learning [7], and the Boltzmann softmax operator and its variants [2].”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “…a similar effect on the value estimates when replacing the Bell-man updates with Double Q-Learning updates, one of many solutions recently proposed to mitigate the negative impact of statistical bias in value function estimation (van Hasselt 2010; Azar et al. 2011; Lee, Defourny, and Powell 2013).”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “Embodiments of this approach are based on dropout Anschel et al. (2017), employing previous Q-function approximations (Ghavamzadeh et al., 2011; An-schel et al., 2017), the linear combination between min and max over the pool of Q-networks (Li & Hou, 2019; Kumar et al., 2019), or the random mixture…”
Dynamic policy programming (2010, methodology): “Very recently, Azar et al. (2011) propose a new variant of Q-learning algorithm, called speedy Q-learning (SQL), which makes use of a different update rule than standard Q-learning of Watkins and Dayan (1992).”
G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “We also include Q ρ -learning, which performs updates as in (4) towards the prior policy ρ. Comparison with Speedy-Q-learning [13] is omitted, since it showed no improvement over vanilla Q-learning in these settings.”
Zap Q-Learning (2017, methodology): “6 [9], Speedy Q-learning [1], and two versions of Zap Q-learning: γn ≡ αn ≡ n−1, and γn ≡ α n ≡ n−0.”
Preference-based Reinforcement Learning with Finite-Time Guarantees (2020, background): “Although value-based RL with a simulator has been well-studied in the literature [3,4], methods like value iteration cannot easily extend to PbRL, because the reward values are hidden.”
Is Q-learning Provably Efficient? (2018, methodology): “When a simulator is available, model-free algorithms [2] (variants of Q-learning) are known to be almost as sample efficient as the best model-based algorithms [3].”
Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction (2020, background): “In addition, [2, 14, 22, 37, 40, 45] studied modified Q-learning algorithms that might potentially improve sample complexities and accelerate convergence.”
Provably Efficient Reinforcement Learning with Linear Function Approximation (2019, background): “See also [23, 6, 7, 24, 36, 44] for a simpliﬁed setting with access to a “simulator” (also called a generative model), which is a strong oracle that allows the algorithm to query arbitrary state-action pairs and return the reward and the next state.”
Provably Efficient Exploration in Policy Optimization (2019, background): “ndition that the state space is already well explored. Such a condition is often implied by assuming either the access to a “simulator” (also known as the generative model) (Koenig and Simmons, 1993; Azar et al., 2011, 2012a,b; Sidford et al., 2018a,b; Wainwright, 2019) or ﬁnite concentratability co”
Bias-corrected Q-learning to control max-operator bias in Q-learning (2013, methodology): “Another approach using two action-value function estimates is Speedy-Q learning [23], which is shown to attain (up to a constant factor) the sample complexity of model-based Q-value iteration (a batch algorithm).”