Active Reinforcement Learning: Observing Rewards at a Cost

By David Krueger, Jan Leike, Owain Evans, John Salvatier. 2020

41 citations recorded by the source corpus.

Active reinforcement learning (ARL) is a variant on reinforcement learning where the agent does not observe the reward unless it chooses to pay a query cost c > 0. The central question of ARL is how to quantify the long-term value of reward information. Even in multi-armed bandits, computing the value of this information is intractable and we have to rely on heuristics. We propose and evaluate several heuristic approaches for ARL in multi-armed bandits and (tabular) Markov decision processes, and discuss and illustrate some challenging aspects of the ARL problem.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tom Everitt
2. Jan Leike
3. Shane Legg
4. Miljan Martic
5. David Krueger
6. Owain Evans
7. Sebastian Schulze
8. Vishal Maini

In Management Science and Operations Research

1. Tom Everitt
2. Owain Evans
3. Sebastian Schulze
4. Jan Leike
5. David Krueger
6. Shane Legg
7. Miljan Martic
8. Vishal Maini

In Reinforcement Learning

1. Tom Everitt
2. Jan Leike
3. Shane Legg
4. David Krueger
5. Miljan Martic
6. Owain Evans
7. Sebastian Schulze
8. Vishal Maini

Citations in the corpus

• Scalable agent alignment via reward modeling: a research direction (2018, methodology): “During training, it can help automate the process of soliciting feedback about the most informative states (Krueger et al., 2016; Schulze & Evans, 2018) using active learning (Settles, 2012).”
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, background): “Techniques from active learning and anomaly detection can be used to detect unfamiliar states [25, 17, 9].”
• Active Reinforcement Learning with Monte-Carlo Tree Search (2018, methodology): “Optimal behaviour in ARL Bandits has a simple form: the agent queries every action up to some point and thereafter never queries (Krueger et al., 2016).”
• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “However, in some references, e.g., [35], [36], “active RL” has a completely different meaning and refers to the RL variant where the agent does not observe the reward unless it pays a query cost to account for the difficulty of collecting reward feedback.”