Policy Search by Dynamic Programming

By J. Bagnell, Sham M. Kakade, Andrew Y. Ng, Jeff Schneider. 2003

202 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, methodology): “While this may lead more quickly to good, possibly globally optimal solutions, such learning processes often prove unstable under function approximation (Boyan and Moore, 1995; Kakade and Langford, 2002; Bagnell et al., 2003) and are considerably more dangerous when applied to real systems where ove”
Dual Policy Iteration (2018, methodology): “5 and classic API policy improvement procedure is that we use η n ’s disadvantage function A η n , i.e., we are performing imitation learning by treating η n as an expert in this iteration [24, 33].”
Approximate Policy Iteration Schemes: A Comparison (2014, methodology): “This algorithm is a natural variation of the Policy Search by Dynamic Programming algorithm (PSDP) of Bagnell et al. (2003), originally proposed to tackle ﬁnite-horizon problems, to the inﬁnite-horizon case; we thus refer to it as PSDP ∞ .”
Agnostic System Identification for Model-Based Reinforcement Learning (2012, neutral): “Methods such as Conservative Policy Iteration (CPI) (Kakade & Langford, 2002) and Policy-Search by Dynamic Programming (PSDP) (Bagnell et al., 2003) learn a policy directly by updating policy parameters iteratively.”
Reinforcement and Imitation Learning via Interactive No-Regret Learning (2014, methodology): “This is simila r in spirit to how Policy Search by Dynamic Programming (PSDP) [4, 28] proceeds, and in some sense, the a lgorithm we present here provides a generalization of PSDP.”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “[22], where a policy search algorithm in the spirit of traditional dynamic programming methods is proposed to learn a non-stationary policy: they learn what should be done in the last time-step and then “back it up” to learn the previous time-step and so on.”
Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning (2018, methodology): “Instead one can leverage the approximate policy iteration idea and minimize the weighted cost over state space using a state distribution ν (Kakade & Langford, 2002; Bagnell et al., 2004):”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “The importance of having a wide and relevant state distribution for RL methods has been pointed out in numerous prior works [10, 43, 44].”
On Oracle-Efficient PAC RL with Rich Observations (2018, methodology): “Inspired by classical RL algorithms, these algorithms approximate g(xh+1) by either bootstrap targets ĝh+1(xh+1) (as in TD methods) or Monte-Carlo estimates of the return using a near-optimal roll-out policy π̂h+1:H (as in PSDP [39]).”
Tree-Based Batch Mode Reinforcement Learning (2005, background): “…the complete generative model assumption of Gordon (footnote 9, page 511). standard supervised classification problems has been developed (see Lagoudakis and Parr, 2003b; Bagnell et al., 2003), taking its roots from the policy iteration algorithm, another classical dynamic programming algorithm.”
Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective (2019, methodology): “(Williams, 1992) and its extensions (Kakade, 2002; Bagnell et al., 2004; Silver et al., 2014; Schulman et al., 2015), compute an estimate of a direction of improvement from sampled trajectories collected by executing a stochastic policy.”
Machine learning of motor skills for robotics (2007, background): “, if the sufficient statistics T (x0:k+1,u0:k) are maintained in a suboptimal filter (Bagnell et al., 2004; Jaakkola, Singh, & Jordan, 1995).”
Variational Policy Search via Trajectory Optimization (2013, unknown): “Several prior methods also propose to improve policy search by using a distribution over high-value states, which might come from a DDP solution [6, 1].”
Reinforcement Learning of POMDPs using Spectral Methods (2016, methodology): “, (Ng and Jordan, 2000),(Baxter and Bartlett, 2001),(Poupart and Boutilier, 2003; Bagnell et al., 2004)).”
Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning? (2019, background): “First, there is Conservative Policy Iteration (Kakade & Langford, 2002), along with Policy Search by Dynamic Programming (PSDP) (Bagnell et al., 2004), and other boosting-style of policy search-based methods Scherrer & Geist (2014); Scherrer (2014), which have guarantees in terms of bounded…”
Analysis of Classification-based Policy Iteration Algorithms (2016, unknown): “Bagnell et al. (2003) introduced an algorithm for learning non-stationary policies in RL.”