Learning from delayed rewards

By C. Watkins. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

A Survey on Neural Architecture Search (2019, methodology): “Zhong et al. (2018) also use a Q-learning-based algorithm to search for network architectures.”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “sible to estimate r(s;a) separately via an unbiased minimal variance estimator like the arithmetic mean and then perform TD updates with the Bellman error using the estimated r(s;a) [103]. Q-learning [140] is an off-policy TD algorithm which is proved to converge [141, 20]. The proofs were later gen”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, background): “…attempting to ﬁnd approximate ﬁxed-point solutions to the Bellman recurrence, leading to the family of temporal-diﬀerence (TD) algorithms including SARSA (Sutton, 1996), Q -learning (Watkins, 1989), and their deep learning variants (Mnih et al., 2015; Hasselt et al., 2016; Wang et al., 2015).”
A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, methodology): “Q-learning [14] is a particular reinforcement learning technique in which the value iteration computations consist of evaluating the corresponding Bellman equation without a model of the environment.”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, methodology): “In particular, our new operators all have a sample-based form, i.e., an analogue to the Q-Learning rule of Watkins (1989).”
Factorized Q-learning for large-scale multi-agent systems (2018, background): “Q-Learning [13, 30, 31] is a model-free o-policy reinforcement learning method that estimates the long-term expected return of executing an action a from a given state s .”
Reinforcement Learning: A Survey (1996, methodology): “The work of the two components of AHC can be accomplished in a uni(cid:12)ed manner by Watkins' Q-learning algorithm (Watkins, 1989; Watkins & Dayan, 1992).”
Multiagent learning using a variable learning rate (2002, background): “Table 4 Algorithm: Opponent Modeling Q-Learning for player i (cid:7) C(s,a − i ) Q (cid:8) s, (cid:12) a ,a (cid:13) (cid:9) the player’s estimates of its opponent’s policy will converge to the true policy.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, positive): “In the Deep Q-Network algorithm (DQN) [Mnih et al., 2015], this value function Q θ is updated in a manner that approximates the optimality operator, through Q-learning [Watkins, 1989]: where l κ deﬁnes the Huber loss [Huber et al., 1964]: but is generally interchangeable with other losses such as me”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “nity (so that each stateaction pair is visited innitely often) and that the sum of the squares of the learning rates is nite (which is required to show that the convergence is with probability one) [46, 45, 47, 48, 49, 50, 51]. The convergence of single-step on-policy RL algorithms, i.e, SARSA (= 0)”
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells (2018, methodology): “Such a simple approach encourages exploration during early stages of search akin to the -greedy strategy often used in the multi-armed bandit problem [42].”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “Arguably the best known model-free online off-policy algorithm is Q-learning (Watkins, 1989; Watkins and Dayan, 1992).”
Reinforcement learning with limited reinforcement: using Bayes risk for active learning in POMDPs (2008, unknown): “Traditional reinforcement learning approaches (Watkins, 1989; Strehl et al., 2006; Even-Dar et al., 2005) to learning in MDP or POMDP domains require a reinforcement signal to be provided after each of the agent’s actions.”
Agent-Based Models and Human Subject Experiments (2004, unknown): “Surprisingly, the predictions of Q—learning models have yet to be compared with data from controlled laboratory experiments with human subjects — a good topic for future research.”
Q-learning and enhanced policy iteration in discounted dynamic programming (2010, methodology): “In this section we consider stochastic iterative algorithms, which are patterned after the classical Q-learning algorithm of Watkins [48] (see also Tsitsiklis [42]), as well as IN F O R M S ho ld s co p yr ig h t to th is ar tic le an d di st rib ut ed th is co py as a co ur te sy to th e au th or (”
A lifelong learning perspective for mobile robot control (1994, methodology): “Notice if Q(s; a) is represented by a lookup-table, the Q-Learning rule has been shown to converge to a value function that yields optimal policies (under certain conditionsconcerning the environment, the exploration scheme and the learning rate) [6, 28].”