Bayesian Reinforcement Learning in Factored POMDPs
By Sammie Katt, Frans Adriaan Oliehoek, Chris Amato. 2018
42 citations recorded by the source corpus.
Model-based Bayesian Reinforcement Learning (BRL) provides a principled solution to dealing with the exploration-exploitation trade-off, but such methods typically assume a fully observable environments. The few Bayesian RL methods that are applicable in partially observable domains, such as the Bayes-Adaptive POMDP (BA-POMDP), scale poorly. To address this issue, we introduce the Factored BA-POMDP model (FBA-POMDP), a framework that is able to learn a compact model of the dynamics by exploiting the underlying structure of a POMDP. The FBA-POMDP framework casts the problem as a planning task, for which we adapt the Monte-Carlo Tree Search planning algorithm and develop a belief tracking method to approximate the joint posterior over the state and model variables. Our empirical results show that this method outperforms a number of BRL baselines and is able to learn efficiently when the factorization is known, as well as learn both the factorization and the model parameters simultaneously.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Biwei Huang
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Bernhard Schölkopf
- Kun Zhang
- Doina Precup
- Chaochao Lu
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “It is also indeed possible to combine the Block MDP and Factored MDP assumptions into a single common formulation (Misra, Liu, Jin, and Langford, 2021) to relax Block MDPs to handle non-stationary environments (Katt, Oliehoek, and Amato, 2019; Sodhani, Meier, Pineau, and Zhang, 2022).”
- AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, methodology): “, 2007) and Factored POMDPs (Katt et al., 2019), but augmented with change factors that represent a low-dimensional embedding of the changes across domains.”