Approximate modified policy iteration and its application to the game of Tetris

By Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, Matthieu Geist. 2015

141 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “(Munos, 2005; Munos and Szepesvári, 2008; Farahmand et al., 2010; Scherrer et al., 2015; Lesner and Scherrer, 2013; Scherrer, 2014) focused on analysing error induced in Bellman projections, under the assumption of an abstract error model.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…errors in approximate dynamic programming (Bertsekas & Tsitsiklis, 1996; Munos, 2005) and approximate policy iteration (Farahmand et al., 2010; Scherrer et al., 2015); for the latter of which there exist well known algorithms that are stable at least with linear function approximation (see…”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “Errors arising from inadequate sampling, distributional shift, and function approximation have been rigorously studied as “error propagation” in approximate dynamic programming (ADP) [5, 28, 11, 34].”
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (2017, methodology): “for a single run of an algorithm, achieved by the CBMPI algorithm (Gabillon et al., 2013; Scherrer et al., 2015), is a mean score of 51 million cleared lines.”
Optimality and Approximation with Policy Gradient Methods in Markov Decision Processes (2019, unknown): “In terms of the concentrability coefficient, there are a notable set of provable average case guarantees for the MD-MPI algorithm [Geist et al., 2019] (see also [Azar et al., 2012, Scherrer et al., 2015]), which are stated in terms of various norms of function approximation error.”
On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift (2019, methodology): “Assuming bounded concentrability coefficient, there are a notable set of provable average case guarantees for the MD-MPI algorithm [Geist et al., 2019] (see also [Azar et al., 2012, Scherrer et al., 2015]), which are stated in terms of various norms of function approximation error.”
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning (2021, background): “Such error have been extensively studied as ”er-ror propagation” in approximate dynamic programming (ADP) (Bertsekas & Tsitsiklis, 1996; Farahmand et al., 2010; Munos, 2003; Scherrer et al., 2015).”
Munchausen Reinforcement Learning (2020, background): “Removing also the blue term, we retrieve the classic AVI [26], of which DQN is an instantiation.”
Is Pessimism Provably Efficient for Offline RL? (2020, result): “In comparison with existing literature (Antos et al., 2007; 2008; Munos and Szepesv´ari, 2008; Farahmand et al., 2010; 2016; Scherrer et al., 2015; Liu et al., 2018; Nachum et al., 2019a;b; Chen and Jiang, 2019; Tang et al., 2019; Kallus and Uehara, 2019; 2020; Fan et al., 2020; Xie and Jiang,…”
A Theory of Regularized Markov Decision Processes (2019, methodology): “The proof is similar to the one of AMPI [Scherrer et al., 2015, Lemma 2], up to the additional term δ k ( π ∗ ) and to the diﬀerent bounded quantity.”
Beyond the One Step Greedy Approach in Reinforcement Learning (2018, methodology): “Proof of Theorem 9 We might follow the steps in (Scherrer & Thiery, 2010; Scherrer et al., 2015).”
A Theoretical Analysis of Deep Q-Learning (2019, background): “1, also known as the error propagation (Munos and Szepesvári, 2008; Lazaric et al., 2016; Scherrer et al., 2015; Farahmand et al., 2010, 2016) in the batch reinforcement”
Mastering the game of Go without human knowledge (2017, methodology): “A more recent instantiation, classificationbased modified policy iteration (CBMPI), also performs policy evaluation by regressing a value function towards truncated rollout values, similar to the value component of AlphaGo Zero; this achieved stateoftheart results in the game of Tetris 38 .”