Optimal learning: computational procedures for bayes-adaptive markov decision processes

By Michael O. Duff, Andrew Barto. 2002

379 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Bayesian Adaptive MDPs (Martin, 1967; Duff, 2002) take this a step further by directly modeling uncertainty in the space of task states.”
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood (2017, background): “, 2014, 2016), and methods which prioritize learning environment dynamics (Duff, 2002) or under-explored states (Kearns and Singh, 2002; Bellemare et al.”
Sample Complexity Bounds of Exploration (2012, background): “Consequently, the agent can simply follow the Bayes optimal policy (Duff, 2002). A major limitation of Bayesian approaches is their high computation complexity. In general, computing the Bayes optimal policy is often intractable, making approximation inevitable (see, e.g., Duff (2002)).”
Point-Based Value Iteration for Continuous POMDPs (2006, methodology): “In this paper, we generalize the results of Duff (2002) and Porta et al. (2005), and describe a framework for optimizing model-based POMDPs in which the state space and/or action and observation spaces are continuous.”
Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints (2014, unknown): “A deterministic policy f : Q∗ → Σ is such that given ρ = q0 . . . qn, f(ρ) = σ only if σ is enabled at qn.”
Scalable Planning and Learning for Multiagent POMDPs (2014, background): “on. This is the task considered in (multiagent) reinforcement learning (RL). BRL methods have become popular because they can provide a principled solution to this exploration/exploitation trade-off (Duff 2002; Engel, Mannor, and Meir 2005; Poupart et al. 2006; Vlassis et al. 2012). POMCP, discussed”
Bayes-Adaptive Simulation-based Search with Value Function Approximation (2014, background): “Below, we first briefly review the Bayesian formulation of optimal decision making under model uncertainty (section 2; please see [8] for additional details).”
Unsupervised Machine Learning for Networking: Techniques, Applications and Research Challenges (2017, methodology): “Basis of re-inforcement learning was also derived by using Bayes theorem [94].”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, methodology): “Here we describe the Bayesian formulation of decision making in an unknown MDP (Martin, 1967; Duff, 2003).”
From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning (2014, background): “Also, following the optimal action of the BAMDP from the current state provides a good exploration-exploitation strategy (which is optimal in a Bayesian sense) [Duff, 2002].”
An intrinsic reward mechanism for efficient exploration (2006, methodology): “Exploration in reinforcement learning has been studied extensively but typically with the objective of maximizing return in an agent’s lifetime, which requires a trade-off between exploration and exploitation, e.g., Duff (2002), Kearns and Singh (1998).”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “By augmenting the belief to the state, a Bayes-Adaptive MDP (BAMDP) can be constructed [7].”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, background): “The Bayes-adaptive MDP formulation (BAMDP) (Duff, 2002; Ross et al., 2008), as well as the closely related hidden parameter MDP (HiP-MDP) (Doshi-Velez & Konidaris, 2016) consider an MDP with unknown parameters governing the reward and dynamics, which we aim to infer online over the course of one…”
Variational methods for Reinforcement Learning (2010, methodology): “where H is the horizon, which can be either finite or infinite, and p(st, at|π) is the marginal of the joint state(1)More generally, one may consider policies which depend on the belief, πa,s,D = p(a|s, p(θ|D), π), similar to the encoding of RL as a POMDP(Duff, 2002), though we leave this case for f”