Q-learning

By Christopher J. C. H. Watkins, Peter S. Dayan. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “The network is trained with a variant of the Q-learning [26] algorithm, with stochastic gradient descent to update the weights.”
Continuous control with deep reinforcement learning (2015, methodology): “Q-learning [9], a commonly used off-policy algorithm, uses the greedy policy µ(s) = argmaxaQ(s, a).”
Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, methodology): “This model learns control strategies us-ing convolutional neural networks, trained with a variant of Q-learning (Watkins and Dayan, 1992).”
Investigating Contingency Awareness Using Atari 2600 Games (2012, methodology): “Their method uses a forward model of the scene to improve the performance of the Q-Learning algorithm (Watkins and Dayan 1992).”
The Arcade Learning Environment: An Evaluation Platform for General Agents (2012, methodology): “Their method uses a forward model of the scene to improve the performance of the Q-Learning algorithm (Watkins & Dayan, 1992).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “Modern techniques for training RL agents can be decomposed into algorithmic choices such as Q-learning (Watkins & Dayan, 1992) or policy gradient (Williams, 1992) and architectural choices for general-purpose function approximators.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “We begin the formulation by reviewing the key elements of Q-learning [43, 44], which uses a notion of hard-max Bellman backup to enable off-policy TD control.”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “In this work, we focus on model-free RL, which includes policy search methods [24], [3], [23] and function approximation methods [25], [26], [14].”
Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “If the learning rate satisﬁes and the interaction itself uses an exploration policy that returns to each state-action pair inﬁnitely many times, then Q is a consistent estimator, converging to Q ∗ with probability 1 [1, 17].”
When to use parametric models in reinforcement learning? (2019, methodology): “…the agent’s predictions in the same way, the resulting algorithm is known as Dyna [Sutton, 1990] – for instance, if predictions v include action values (normally denoted with q) and we update using Q-learning [Watkins, 1989, Watkins and Dayan, 1992], we obtain Dyna-Q [Sutton and Barto, 2018].”
Value Prediction Network (2017, background): “3 describes how to train VPN in a Q-Learning-like framework [38].”
Prioritized Experience Replay (2015, methodology): “Furthermore, we ﬁnd that the boost from prioritized experience replay is complementary to the one from introducing Double Q-learning into DQN: performance increases another notch, leading to the current state-of-the-art on the Atari benchmark (see Figure 3).”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “Specifically, DQN [18] combined Q-learning [29] with a CNN and achieved state-of-the-art performance on many Atari games.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…fully minimize the difference between the left-hand side and right-hand side of the above equation at each iteration, commonly referred to as fitted Q-iteration (Ernst et al., 2005), and variants that take a single gradient step, such as the original Q-learning method (Watkins and Dayan, 1992).”
Deep learning in neural networks: An overview (2014, neutral): “…Williams, 1996; Prokhorov &Wunsch, 1997; Rummery &Niranjan, 1994; Santamaría, Sutton, & Ram, 1997; Schwartz, 1993; Singh, 1994; Sutton & Barto, 1998; Sutton, Szepesvári, & Maei, 2008; Tsitsiklis & van Roy, 1996; van Hasselt, 2012; Watkins, 1989; Watkins & Dayan, 1992; Wiering & Schmidhuber, 1998b).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Q-learning [127] is an off-policy TD algorithm which is proved to converge [128, 14].”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “Other work addressed the problem of nonsmoothness of decision rules as well as nonregularity of the parameter estimation in traditional RL methods by proposing various extensions over the Q-learning procedure to increase the robustness of learning.”