A Convergent Form of Approximate Policy Iteration

By Theodore J. Perkins, Doina Precup. 2002

122 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Maximum a Posteriori Policy Optimisation (2018, background): “The most prominent examples for this are formed by work on Boltzmann exploration such as Kael-bling et al. (1996); Perkins & Precup (2002); Sutton (1990); O’Donoghue et al. (2017), which can be connected back to soft Q-learning (and thus to our approach) as shown in Haarnoja et al. (2017).”
Dynamic policy programming (2010, neutral): “Examples of such approximate dynamic programming (ADP) methods are approximate policy iteration (API) and approximate value iteration (AVI) (Bertsekas, 2007; Lagoudakis and Parr, 2003; Perkins and Precup, 2002; de Farias and Roy, 2000).”
A unified view of entropy-regularized Markov decision processes (2017, methodology): “Various approximations of this update have been long studied in the RL literature—see, e.g., (Littman and Szepesvári, 1996) (with additional discussion by (Asadi and Littman, 2016)), (Perkins and Precup, 2002; Ruszczyński, 2010; Petrik and Subramanian, 2012; Fox et al., 2016).”
Algorithms for Reinforcement Learning (2010, methodology): “Perkins and Precup (2003) analyze this choice with linear function approximation and when the behavior and target policies are the same.”
An analysis of reinforcement learning with function approximation (2008, methodology): “In our main result, we recover the essence of the result in ( Perkins & Precup, 2003 ), although in a somewhat dierent setting.”
PAC Reinforcement Learning with Rich Observations (2016, background): “Several authors have studied linear or more general function approximation (See [5, 24, 28]), but none of these results give finite sample bounds, as they do not address the exploration question.”
Reinforcement Learning for RoboCup Soccer Keepaway (2005, background): “…to the optimal policy in all cases, but several lines of evidence suggest that it is near a good solution (Gordon, 2001; Tsitsiklis & Van Roy, 1997; Sutton, 1996) and recent results show that it does indeed con- verge, as long as the action-selection policy is continuous (Perkins & Precup, 2003).”
Policy Gradient Methods for Off-policy Control (2015, unknown): “A I] 1 3 D ec 2 01 methods such as [8].”
Boltzmann Exploration Done Right (2017, background): “Within Reinforcement Learning [23, 25], exponential weighting schemes are broadly used for balancing exploration and exploitation, and are equivalently referred to as Boltzmann, Gibbs, or softmax exploration policies [22, 14, 24, 19].”
Error Bounds for Approximate Policy Iteration (2003, background): “Extend results to other policy improvement operators: - Optimistic PI [Bertsekas & Tsitsiklis, 1996] - Conservative PI [Kakade & Langford, 2002] - Smooth PI [Perkins & Precup, 2002] References Bertsekas & Tsitsiklis, 1996 : Neuro-Dynamic Programming.”
Safe Policy Iteration (2013, background): “A few approaches (Perkins & Precup, 2002; Kakade & Langford, 2002; Wagner, 2011; Azar et al., 2012) to this problem, instead of iterating on a sequence of greedy policies computed on approximated action– value functions, propose converging algorithms that exploit smaller updates in the space of…”
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (2015, methodology): “We further study the convergence properties of the proposed Actor-Mimic under a framework similar to (Perkins & Precup, 2002).”
A New Softmax Operator for Reinforcement Learning (2016, background): “1 and β = 16 .”
Experience Replay for Real-Time Reinforcement Learning Control (2012, background): “Approximate SARSA converges— under appropriate conditions—both when the policy is changed after every sample [20] and when it is changed only after a large number of updates [23].”