Approximately Optimal Approximate Reinforcement Learning

By Sham M. Kakade, J. Langford. 2002

1,346 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Proximal Policy Optimization Algorithms (2017, background): “TRPO maximizes a “surrogate” objective The superscript CPI refers to conservative policy iteration [KL02], where this objective was proposed.”
Trust Region Policy Optimization (2015, methodology): “The ﬁrst proof extends Kakade and Langford’s result using the fact that the random variables from two distributions with total variation divergence less than α can be coupled, so that they are equal with probability 1 − α .”
Constrained Policy Optimization (2017, methodology): “This result, which is of independent interest, tightens known bounds for policy search using trust regions (Kakade & Langford, 2002; Pirotta et al., 2013; Schulman et al., 2015), and provides a tighter connection between the theory and practice of policy search for deep RL.”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “…expected improvement η ( π ) can be expressed in terms of the advantage A µ ( s , a ) = R µ s , a − V µ ( s ) with respect to the sampling policy µ (Kakade & Langford, 2002; Schulman et al., 2015): where R µ s , a denotes the return obtained by performing action a in state s and following µ for…”
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2010, methodology): “Another approach called SMILe (Ross and Bagnell, 2010), similar to SEARN (Daumé III et al., 2009) and CPI (Kakade and Langford, 2002), trains a stationary stochastic policy (a ﬁnite mixture of policies) by adding a new policy to the mixture at each iteration of training.”
Search-based structured prediction (2009, neutral): “Then, for all D , with c max = E ( x, c ) ∼D max i c i (with ( x, c ) as in Def 1): Proof The proof largely follows the proofs of Lem 6.1 and Theorem 4.1 for conservative policy iteration [23].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “For example, by constraining how much the learned policy π(a|s) differs from the behavior policy πβ(a|s), we can bound state distributional shift (Kakade and Langford, 2002; Schulman et al., 2015a).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Kakade and Langford [28] proposed the CPI algorithm, that improves a policy conservatively in online RL.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Our work connects to policy methods with conservative updates (Kakade & Langford, 2002), such as trust region (Schulman et al., 2015; Achiam et al., 2017) and information-theoretic methods (Peters & M ¨ ulling, 2010; Van Hoof et al., 2017) which aim to keep an updated policy similar to the previous…”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “TRPO extends the policy improvement bound in the mixture policies setting given by Kakade and Langford (2002) to general stochastic policies, and shows that where This means that by performing the following optimization (here we denote L θ old ( θ ) := L π θ old ( π θ ) ) with parameterized…”
Reinforcement learning in robotics: A survey (2013, background): “While this may lead more quickly to good, possibly globally optimal solutions, such learning processes often prove unstable under function approximation (Boyan and Moore, 1995; Kakade and Langford, 2002; Bagnell et al., 2003) and are considerably more dangerous when applied to real systems where ove”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “More recently, Thomas et al. (2015a) incorporate lower conﬁdence bounds with IS in approximate policy iteration to ensure that the computed policy meets a minimum value guarantee.”
Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation (2018, background): “…ratio that is clipped to the range of [1 − (cid:15), 1 + (cid:15) ] , making the L CLIP ( θ ) a lower bound of the conservative policy iteration objective [17], ˆ A t is the estimated advantage function which involves a learned value function V ω ( · ) to reduce the variance of estimation.”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “To analyze the bias for this gradient expression, we ﬁrst introduce ˜ J ( π, ˜ π ) , a local approximation to J ( π ) , which has been used in prior theoretical work (Kakade & Langford, 2002; Schulman et al., 2015).”
MOReL : Model-Based Offline Reinforcement Learning (2020, neutral): “Note that the policy is being learned in the P-MDP, and as a result we observe a clear monotonic learning curve for value in the P-MDP, consistent with the monotonic improvement theory of policy gradient methods [75, 76].”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “The idea of conservative policy optimization can be traced back to Kakade & Langford (2002).”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, neutral): “This idea might seem similar to Conservative Policy Iteration (CPI), Trust Region Policy Optimization (TRPO), or Proximal Policy Optimization (PPO) in that it allows changes in the policy under a proximity regularization to the old policy (Kakade and Langford, 2002; Schulman et al., 2015, 2017).”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “Kakade and Langford [23] shows that large policy updates can be destructive, and propose a conservative policy iteration scheme which constrains actor updates to be small for provably convergent learning.”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “For any policy π and dynamical models M, (cid:99) M , we have that The proof is reminiscent of the telescoping expansion in Kakade & Langford (2002) (c.f. Schulman et al. (2015a)) for characterizing the value difference of two policies, but we apply it to deal with the discrepancy between models.”
Dual Policy Iteration (2018, methodology): “We show that by integrating local model-based search with learned local dynamics into policy improvement via an imitation learning-style update, our algorithm is substantially more sampleefficient than classic API algorithms such as CPI [5], as well as more recent actor-critic baselines [23], albeit”