Active Reinforcement Learning with Monte-Carlo Tree Search
By Sebastian Schulze, Owain Evans. 2018
20 citations recorded by the source corpus.
Active Reinforcement Learning (ARL) is a twist on RL where the agent observes reward information only if it pays a cost. This subtle change makes exploration substantially more challenging. Powerful principles in RL like optimism, Thompson sampling, and random exploration do not help with ARL. We relate ARL in tabular environments to Bayes-Adaptive MDPs. We provide an ARL algorithm using Monte-Carlo Tree Search that is asymptotically Bayes optimal. Experimentally, this algorithm is near-optimal on small Bandit problems and MDPs. On larger MDPs it outperforms a Q-learner augmented with specialised heuristics for ARL. By analysing exploration behaviour in detail, we uncover obstacles to scaling up simulation-based algorithms for ARL.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computational Theory and Mathematics
In Management Science and Operations Research
In Reinforcement Learning
- Jan Leike
- Tom Everitt
- David Krueger
- Shane Legg
- Miljan Martic
- Vishal Maini
- Sergey Levine
- Iason Gabriel
Citations in the corpus
- Scalable agent alignment via reward modeling: a research direction (2018, methodology): “During training, it can help automate the process of soliciting feedback about the most informative states (Krueger et al., 2016; Schulze & Evans, 2018) using active learning (Settles, 2012).”