Regularized Policy Iteration

By Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvari, Shie Mannor. 2008

148 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “Regularization in RL Some early attempts to learn RL function approximators used `2 regularization of the Q (Farahmand et al., 2008; Yan et al., 2017) function.”
Batch Policy Learning under Constraints (2019, methodology): “(Munos, 2003; 2007; Munos & Szepesvári, 2008; Antos et al., 2008a;b; Lazaric et al., 2010; 2012; Farahmand et al., 2009; Maillard et al., 2010) Let P be the operator acting on f : X × A 7→ R s.”
Dynamic policy programming (2010, background): “We also ﬁx the total number of samples, per state-action, to 1 × 10 5 samples for all problems and algorithms.”
Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view (2010, methodology): “We focus on two popular methods: the computation of the projected Temporal Difference fixed point (TD(0), TD for short), which Antos et al. (2008); Farahmand et al. (2008); Sutton et al. (2009) have recently presented as the minimization of the mean-square projected Bellman Appearing in Proceedings…”
A Kernel Loss for Solving the Bellman Equation (2019, background): “Our work is also related to a line of interesting work on Bellman residual minimization (BRM) based on nested optimization (Antos et al., 2008; Farahmand et al., 2008, 2016; Hoffman et al., 2011; Chen & Jiang, 2019).”
Finite-sample analysis of least-squares policy iteration (2012, background): “Oﬀ-policy LSTD. Yu and Bertsekas (2010) derived new bounds for projected linear equations substituting the 1 √ 1 − γ 2 term in front of the approximation error with a much sharper term depending on the spectral radius of some matrices deﬁned by the problem.”
Manifold Regularization for Kernelized LSTD (2017, methodology): “[9] introduce an `2 regularization extension to kernelized LSTD [10], termed Regularized LSTD (REG-LSTD), featuring better control of the complexity of the function approximator through regularization, and mitigating the burden of selecting basis functions through kernel machinery.”
Error Propagation for Approximate Policy and Value Iteration (2010, unknown): “Also our results indicate that the contribution of the approximation/Bellman error to the performance loss is more prominent in the later iterations of API/AVI, and the effect of an error term in the earlier iterations decays exponentially fast.”
Regularization and feature selection in least-squares temporal difference learning (2009, background): “For instance, (Farahmand et al., 2009) consider a regularized version of TD-based policy iteration algorithms, but only speci.cally consider l2 regularization their speci.c scheme for l2 regularization varies slightly from ours, but the general idea is quite similar to the l2 regularization…”
Non-Parametric Approximate Linear Programming for MDPs (2011, background): “Farahmand et al. (2009) are notable for including sample complexity results and max-norm error bounds, but their bounds depend upon difficult to measure quantities, such as concentrability coefficients.”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “…have been efforts to avoid the double-sampling problem for other approaches such as the projected fixpoint methods or Bellman residual minimization (Farahmand et al., 2008), it is still an open question whether the bias of the residual-gradient algorithm with single samples can be removed by…”
Reinforcement Learning under Model Mismatch (2017, methodology): “The main idea of [24] is to devise stochastic gradient algorithms to minimize the following loss function called the mean square projected Bellman error ( MSPBE ) also studied in [1, 12].”
Regularization Matters in Policy Optimization (2019, background): “There have been many prior works studying the theory of regularization in policy optimization (Farahmand et al., 2009; Neu et al., 2017; Zhang et al., 2020).”
Generalization and Regularization in DQN (2018, background): “The few existing studies often focus on single-task settings with linear function approximation (e.g., Farahmand et al., 2008; Kolter and Ng, 2009).”
On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes (2012, methodology): “he actual proof of this conjecture is left for future work. Important recent works of the literature involve studying performance bounds when the errors are controlled in Lp norms instead of max-norm [19, 20, 21, 1, 8, 18, 17] which is naturalwhen supervised learning algorithms are used to approxima”
Kernelized value function approximation for reinforcement learning (2009, background): “Extremely recent work by Farahmand et al. (2008) introduces a regularized, kernelized version of LSTD.”
Alignment for Advanced Machine Learning Systems (2020, background): “Regularization helps ML systems prevent overﬁtting, and has been applied to the problem of learning value functions for policies in order to learn less-extreme policies that are more likely to generalize well (Farahmand et al. 2009).”