Finite time bounds for sampling based fitted value iteration

By Csaba Szepesvari, R´emi Munos. 2005

133 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Nan Jiang
3. Csaba Szepesvari
4. Sham M. Kakade
5. Alekh Agarwal
6. Lihong Li
7. R´emi Munos
8. Ruslan Salakhutdinov

In Statistics and Probability

1. Nan Jiang
2. Alekh Agarwal
3. Sham M. Kakade
4. Emma Brunskill
5. Csaba Szepesvari
6. Ruosong Wang
7. Tengyang Xie
8. Shie Mannor

In Reinforcement Learning

1. Sergey Levine
2. Nan Jiang
3. Emma Brunskill
4. Lihong Li
5. Alekh Agarwal
6. Csaba Szepesvari
7. Sham M. Kakade
8. R´emi Munos

In Statistical Learning Theory

1. Nan Jiang
2. Emma Brunskill
3. Alekh Agarwal
4. Ruosong Wang
5. Csaba Szepesvari
6. Sham M. Kakade
7. R´emi Munos
8. Tengyang Xie

Citations in the corpus

• Information-Theoretic Considerations in Batch Reinforcement Learning (2019, background): “…they work is central to our understanding of RL. Existing works that analyze error propagation and finite sample behavior of ADP methods (Munos, 2003; Szepesv´ari & Munos, 2005; Antos et al., 2008; Munos & Szepesv´ari, 2008; Tosatto et al., 2017) have provided us with a decent understanding: To…”
• Agnostic System Identification for Model-Based Reinforcement Learning (2012, neutral): “, policy gradient (Williams, 1992), fitted value iteration (Szepesvári, 2005) or iLQR (Li & Todorov, 2004).”
• What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, background): “…a real issue in offline RL, and sample-efficient offline RL is impossible unless the distribution shift is sufficiently low, i.e., ∏H h=1Ch is bounded, or stronger representation condition such as policy completeness is assumed as in prior works [Szepesvári and Munos, 2005, Chen and Jiang, 2019].”
• Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “, maxπ C [26], [33], [57]–[61] and some definitions are more complex and stronger assuming a bounded ratio per time step [24], [25], [62]–[64].”
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift (2019, unknown): “Williams [1992], Sutton et al.”
• Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval (2015, methodology): “the fitted value iteration (FVI) [357] algorithm) using a corpus of historical data of user interactions, or simulated users generated based on some of such data [358].”
• Provably Good Batch Reinforcement Learning Without Great Exploration (2020, methodology): “Our algorithms need the hyperparameter b to trade off conservatism of a large b (where the algorithm stays at its initialization in the limit) and unfounded optimism of b = 0 (classical FQI/FPI).”
• Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems (2006, background): “Although the parallel sampling model is not a realistic model it is often considered in theory as a relaxation of the MDP model which still captures many important aspects of the original problem; see Kearns and Singh (2002), Szepesvri and Munos (2005).”