An analysis of model-based Interval Estimation for Markov Decision Processes

By Alexander L. Strehl, Michael L. Littman. 2008

651 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Most approaches for exploration focus on the tabular case and generally learn models of the environment (e.g., Brafman & Tennenholtz, 2002; Kearns & Singh, 2002; Strehl & Littman, 2008).”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “For example, Model-based Interval Estimation with Exploration Bonuses (MBIE-EB; Strehl and Littman, 2008) solves the augmented Bellman equation involving the empirical reward ˆ R , the empirical transition function ˆ P , and an exploration bonus proportional to N ( x, a ) − 1 / 2 .”
Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models (2015, methodology): “We are interested in ﬁnding a policy π : S × A → [0 , 1] that maximizes the expected reward over all time.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, neutral): “Uncertainty estimates in deep reinforcement learning have generally been used to encourage exploration (Dearden et al., 1998; Strehl & Littman, 2008; O’Donoghue et al., 2018; Azizzadenesheli et al., 2018).”
Self-Imitation Learning (2018, methodology): “Many existing methods use some notion of curiosity or uncertainty as a signal for exploration (Schmidhuber, 1991; Strehl & Littman, 2008).”
MOPO: Model-based Offline Policy Optimization (2020, background): “Uncertainty quantiﬁcation, a key ingredient to our approach, is critical to good performance in model-based RL both theoretically [62, 73, 43] and empirically [11, 6, 49, 38, 7], in optimal control [61, 2, 33].”
Exploration by Random Network Distillation (2018, background): “Count-based exploration bonuses are a natural and effective way to do exploration (Strehl & Littman, 2008) and a lot of work has studied how to tractably generalize count bonuses to large state spaces (Bellemare et al., 2016; Fu et al., 2017; Ostrovski et al., 2017; Tang et al., 2017; Machado et…”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, methodology): “We consider a small tabular MDP called DeepSea [39] shown in Figure 1, which can be thought of as an unrolled version of the RiverSwim environment [52].”
Emergent Tool Use From Multi-Agent Autocurricula (2019, methodology): “One example is count-based exploration, where agents are incentivized to reach infrequently visited states by maintaining state visitation counts [37, 38, 39] or density estimators [40, 41].”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, methodology): “See Appendix B for all the hyperparameter values used in the experiments.”
Behaviour Suite for Reinforcement Learning (2019, background): “Examples of this include ‘RiverSwim’ for exploration (Strehl & Littman, 2008) or ‘Taxi’ for temporal abstraction (Dietterich, 2000).”
PAC Bounds for Discounted MDPs (2012, methodology): “This assumption aside, the new upper bound improves on the previously best known bound of Auer (2011). If the assumption is dropped then the new proof can be used to construct an algorithm that is better than the bound of Auer (2011) in terms of 1/(1− γ), but worse in |S|.”
(More) Efficient Reinforcement Learning via Posterior Sampling (2013, methodology): “Using optimism with KL-divergence instead of L1 balls has also shown improved performance over UCRL2 [22], but its regret remains orders of magnitude more than PSRL on RiverSwim.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Curiosity or uncertainty has been used as a signal for exploration [33, 35, 23, 29].”
Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning (2015, background): “The proof of Theorem 1 is quite long and involved, but builds on similar techniques for sample-complexity bounds in reinforcement learning (see e.g. Brafman and Tennenholtz [3], Strehl and Littman [16]).”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, background): “A number of prior works have been proposed to guide exploration based on criteria such as intrinsic motivation (Schmidhuber, 2008; Stadie et al., 2015), state-visitation counts (Strehl & Littman, 2008; Bellemare et al., 2016), and optimism in the face of uncertainty (Brafman & Tennenholtz, 2003).”