A theoretical and empirical analysis of Expected Sarsa

By Harm van Seijen, Hado van Hasselt, Shimon Whiteson, Marco A. Wiering. 2009

245 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, neutral): “The outcome is an algorithm reminiscent of Expected SARSA (Van Seijen et al., 2009), where the value estimate is instead learned off-policy and the noise added to the target policy is chosen independently of the exploration policy.”
Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Such awareness to the cost of exploration is usually attributed to on-policy algorithms, such as SARSA [2, 4] and Expected-SARSA [15, 16].”
Dueling Network Architectures for Deep Reinforcement Learning (2015, methodology): “More speciﬁcally, given a behavior policy π , we seek to estimate the state-action value Q π ( · , · ) by optimizing the sequence of costs of equation (4), with target The above update rule is the same as that of Expected SARSA (van Seijen et al., 2009).”
Meta-Gradient Reinforcement Learning (2018, methodology): “A large family of RL algorithms [Sutton, 1988, Rummery and Niranjan, 1994, van Seijen et al., 2009, Sutton and Barto, 2018], including several state-of-the-art deep RL algorithms [Mnih et al., 2015, van Hasselt et al., 2016, Harutyunyan et al., 2016, Hessel et al., 2018, Espeholt et al., 2018], are…”
Q(λ) with Off-Policy Corrections (2016, background): “Given a sample experience x, a, r, x′, a′, SARSA(0) [12] updates its Q-function estimate at kth iteration as follows: Qk+1(x, a)← Qk(x, a) + αkδ, δ = r + γQk(x ′, a′)−Qk(x, a), where δ is the TD-error, and (αk)k∈N a sequence of nonnegative stepsizes.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Furthermore, Van Seijen [53] has proven convergence for Expected SARSA (see Section 3.”
Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, result): “To further clarify, if we could calculate R, replacing all rewards in the trajectories with R will result in a lower variance policy update, similar to expected SARSA [34].”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “Any standard RL update can be used in the outer loss, such as Q-learning [41], Q( λ ), or (Expected) Sarsa [26, 37].”
Flexible Option Learning (2021, background): “As such, they are reminescent of generalizations of standard RL algorithms, such Expected SARSA [van Seijen et al., 2009] and Expected Policy Gradients [Ciosek and Whiteson, 2018].”
Q( λ ) with Oﬀ-Policy Corrections (background): “…can imagine considering expectations over action-values at the corresponding states E π Q ( x t , · ), in place of the value of the sampled action Q ( x t , a t ), i.e.: This is the one-step update for General Q-Learning [18], which is a generalization of Expected SARSA [20] to arbitrary policies.”
G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, background): “Top: G-learning converges to the optimal value faster than both Q-learning and Expected-SARSA.”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “Recently, an extension to more than one opponent, multiagent ToM (MToM), was proposed by Van der Osten et al. (2017).”
Expected Policy Gradients (2017, background): “Typically, (1) is approximated from samples from a trajectory τ of length T : (2) If the policy is deterministic, we can use deterministic policy gradients (Silver et al., 2014) instead: ( This update is then approximated using samples: Since the policy is deterministic, the problem of exploration i”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “Examples of on-policy TD-learning algorithms include SARSA [20], Expected SARSA [21], Actor-Critic [22], and its deep neural network extension called A3C [23].”
Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018, methodology): “[29], we determine the variance of the standard discounted Bellman equation: [30]: Gγt = rt + γV γ π (st+1).”