Dual Representations for Dynamic Programming and Reinforcement Learning

By Tao Wang, Michael H. Bowling, Dale Schuurmans. 2007

46 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Information Systems

Citations in the corpus

GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “To make the optimization tractable and address a double sampling issue, Zhang et al. (2020a) rewrite φ(x) as φ(x) = maxf∈R xf − (1)2φ(f), where φ∗ is the Fenchel conjugate of φ, and use the interchangeability principle for interchanging maximization and expectation (Shapiro et al.”
Compress and Control (2014, methodology): “ng the stationary distribution of a particular kind of augmented Markov chain describing the system, from which we arXiv:1411.5326v1 [cs.AI] 19 Nov 2014 can approximate a type of dual representation (Wang, Bowling, and Schuurmans, 2007; Wang et al., 2008) of the value function. Using this technique,”
Dynamic policy programming (2010, background): “…policy (Farahmand et al., 2010; Thiery and Scherrer, 2010; Munos, API and AVI are expressed in terms of the supremum, with respect to (w.r.t.) the number of iterations, of the approximation errors: where γ denotes the discount factor, k · k is the ℓ ∞ -norm w.r.t. the state-action pair ( x, a ).”
Eigenoption Discovery through the Deep Successor Representation (2017, background): “It can be seen as the dual approach to dynamic programming and to value-function based methods in reinforcement learning (Wang et al., 2007).”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “…algorithm, Gradient Emphasis Learning (GEM), to approximate the emphasis in COF-PAC, inspired by Gradient TD methods (GTD, Sutton et al. 2009b;a), Emphatic TD methods (ETD, Sutton et al. 2016), and reversed TD methods (Wang et al., 2007; 2008; Hallak & Mannor, 2017; Gelada & Bellemare, 2019).”
The Value Function Polytope in Reinforcement Learning (2019, background): “Contrary to the primal form, any feasible discounted state action visit frequency distribution maps to an actual policy (Wang et al., 2007).”
Dual Temporal Difference Learning (2009, background): “Experimental results on random MDPs and the mountain car problems suggest that the dual algorithms seem to demonstrate empirical benefits over standard primal algorithms.”
Stable Dual Dynamic Programming (2007, background): “Dual representations maintain an explicit representation of visit distributions as opposed to value functions [4].”