Error Propagation for Approximate Policy and Value Iteration

By Amir-massoud Farahmand, R´emi Munos, Csaba Szepesvari. 2010

275 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Since the error in policy performance compounds with a factor that depends on 1/(1− γ)(2) ≈ H(2) (Farahmand et al., 2010; Kumar et al., 2019a), a small divergence from the behavior policy at each step can give rise to a policy that diverges away from the behavior distribution and performs very poorl”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “[13] Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos.”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, background): “…those from deterministic expert policies, are particularly problematic for oﬄine RL algorithms and may cause divergence both empirically [Fujimoto et al., 2018, Kumar et al., 2019] and theoretically [Munos, 2003, Farahmand et al., 2010, Kumar et al., 2019, Agarwal et al., 2019a, Du et al., 2020].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “(Munos, 2005; Munos and Szepesvári, 2008; Farahmand et al., 2010; Scherrer et al., 2015; Lesner and Scherrer, 2013; Scherrer, 2014) focused on analysing error induced in Bellman projections, under the assumption of an abstract error model.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “In particular, several papers have theoretically analyzed the accumulation of bootstrapping errors in approximate dynamic programming (Bertsekas & Tsitsiklis, 1996; Munos, 2005) and approximate policy iteration (Farahmand et al., 2010; Scherrer et al., 2015); for the latter of which there exist well”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “Errors arising from inadequate sampling, distributional shift, and function approximation have been rigorously studied as “error propagation” in approximate dynamic programming (ADP) [5, 28, 11, 34].”
Information-Theoretic Considerations in Batch Reinforcement Learning (2019, methodology): “Prior works have used more sophisticated deﬁnitions (Farahmand et al., 2010).”
Datasets for Data-Driven Reinforcement Learning (2020, background): “…those from deterministic expert policies, are particularly problematic for offline RL algorithms and may cause divergence both empirically [Fujimoto et al., 2018, Kumar et al., 2019] and theoretically [Munos, 2003, Farahmand et al., 2010, Kumar et al., 2019, Agarwal et al., 2019a, Du et al., 2020].”
Analysis of Classification-based Policy Iteration Algorithms (2016, result): “classification-based API algorithms (Lagoudakis and Parr, 2003b; Fern et al., 2004) is in weighting each classification error by its actual regret, i.e., the difference between the actionvalues of the greedy action and the action selected by DPI. Our results extend the only theoretical analysis of a”
Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, background): “, approximation/estimation errors, and incorporating them is routine [25, 26, 27].”
Approximate Dynamic Programming for Two-Player Zero-Sum Markov Games (2015, background): “Bounds in L∞ can be found in (Bertsekas, 1995) while Lp-norm ones were published in (Munos & Szepesvári, 2008) and (Farahmand et al., 2010).”
Boosted Curriculum Reinforcement Learning (2022, result): “The use of AVI algorithms in this curriculum RL setting consists in starting from an arbitrary estimate Q 01 (where the subscript indicates the position of the task in the curriculum) and approximating the application of the task-speciﬁc optimal Bellman operator for the current task We assume that t”