When the Best Move Isn't Optimal: Q-learning with Exploration

By George H. John. 1994

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Such awareness to the cost of exploration is usually attributed to on-policy algorithms, such as SARSA [2, 4] and Expected-SARSA [15, 16].”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “the sample determines which (s0;a0) is chosen. The TD methods differ in how they select a0. SARSA [97] selects a0by sampling from the policy: E s0;a0[^q ˇ(s0;a0)] ˇq^ˇ(S t+1;A t+1) and expected SARSA [55] averages over selections E s0;a0[^q ˇ(s0;a0)] ˇ X a ˇ(ajS t+1) ^qˇ(S t+1;a): It is possible to ”
A New Softmax Operator for Reinforcement Learning (2016, background): “When learning “on policy”, exploration steps can (Rummery & Niranjan, 1994) and perhaps should (John, 1994) become part of the value-estimation process itself.”
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms (2000, methodology): “Note that if a GLIE learning policy is used with the Q-learning update rule, one gets convergence to both the optimal Q-value function and an optimal policy. This begins to address a significant outstanding question in the theory of reinforcement learning: How do you a learn a policy that achieves h”
G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, background): “Note that the advantage of G-learning over Q-learning is more manifest in the latter, noisier cases. of on-policy algorithms such as SARSA or Expected- SARSA [16] [39] over off-policy learning approaches such as Q-learning.”
A Generalized Reinforcement-Learning Model: Convergence and Applications (1996, background): “.3 RISK-SENSITIVE MODELS Heger (1994) described an optimality criterion for mdps in which only the worst possible value of the next state makes a contribution to the value of a state.”
MDPs: Learning in Varying Environments (2002, background): “…MDP model is that a wide range of models, e.g., Markov games (Littman, 1994), alternating Markov games (Boyan, 1992), discounted expected-reward MDPs (Watkins and Dayan, 1992), risk-sensitive MDPs (Heger, 1994), exploration-sensitive MDPs (John, 1994) can be discussed in this unified framework.”
A Generalized Reinforcement-Learning Model : Convergence and Applications (2010, background): “…reinforcement-learning scenarios are specia, l ca. ses of this genera, lir,ed "MDP model includ ing computing the expected return of a fixed pol icy (Sutton: 1988), finding the optimal risk-averse pol icy (Heger, 1994): a, nd finding the optimal exploration sensitive policy (John, 1995).”
A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms (1999, background): “Other extensions of the results of this section are to the case in which the spreading function s decays to one that guarantees convergence to an optimal Q function, and the case in which learned values are a function of the chosen exploratory actions (the so-called sarsa algorithm) (John 1994; Rumm”
Apprenticeship Learning About Multiple Intentions (2011, background): “This approach makes the likelihood (inﬁnitely) diﬀeren-tiable, although, in practice, other mappings could be used.”
Algorithms for Sequential Decision Making (1996, methodology): “In John's formulation, the agent is forced to adopt a policy from 58a restricted set; in one example, the agent must choose a stochastic stationary policythat selects actions at random 5% of the time.”