Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches

By Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, J. Langford. 2018

245 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Information-Theoretic Considerations in Batch Reinforcement Learning (2019, background): “…a restricted value-function class will not work Sutton & Barto (2018, Section 11.6) proposes a clever way to prevent the learner to be model-based for linear function approximation, and a closely related definition is recently given by Sun et al. (2019) that applies to arbitrary function classes.”
Representation Learning for Online and Offline RL in Low-rank MDPs (2021, methodology): “Following prior work (Jiang et al., 2017; Sun et al., 2019), we assume trajectory reward is normalized, i.”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “In single-agent RL, there exists a significant amount of research regarding the model-based RL methods; see for instance [199]; however, their extension to MARL has not been explored widely.”
FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs (2020, methodology): “Sun et al. (2019) extend the deﬁnition to model-based approaches, with the notion of Witness rank.”
Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms (2021, background): “, low witness rank [36]) while [41] can.”
On Value Functions and the Agent-Environment Boundary (2019, background): “This corresponds to the state compression scheme, s 7→ {f(s, a) : a ∈ A, f ∈ F} [Sun et al., 2019], but such a definition is very brittle, as even the slightest numerical changes in F might cause the boundary to change significantly.”
Combating the Compounding-Error Problem with a Multi-step Model (2019, background): “…is well-understood in theory in settings such as value-function estimation [Azar et al., 2012], exploration in discounted and finite-horizon Markov decision processes (MDPs) [Szita and Szepesvári, 2010; Dann and Brunskill, 2015], and exploration in contextual decision processes [Sun et al., 2018].”
Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning? (2019, background): “Recently, Witness rank, a generalization of Bellman rank to model-based methods, is studied in Sun et al. (2019).”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, result): “Considering recent progresses in model-based RL, especially its provable advantages over model-free ones in certain regimes [378, 379], it is worth generalizing these results to MARL to improve its sample efficiency.”
Provably efficient representation selection in Low-rank Markov Decision Processes: from online to offline RL (2021, background): “…reinforcement learning, existing works on representation learning [Jiang et al., 2017, Agarwal et al., 2020, Modi et al., 2021, Uehara et al., 2021, Sun et al., 2019, Du et al., 2021] often assumed that the transition dynamic can be represented as a linear function of an unknown representation,…”
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States (2021, background): “As optimal policies for factored MDPs do not factorize, therefore, a model-free approach is unlikely to succeed (Sun et al., 2019).”