Making Sense of Reinforcement Learning and Probabilistic Inference

By Brendan O'Donoghue, Ian Osband, Catalin Ionescu. 2020

56 citations recorded by the source corpus.

Reinforcement learning (RL) combines a control problem with statistical estimation: The system dynamics are not known to the agent, but can be learned through experience. A recent line of research casts 'RL as inference' and suggests a particular framework to generalize the RL problem as probabilistic inference. Our paper surfaces a key shortcoming in that approach, and clarifies the sense in which RL can be coherently cast as an inference problem. In particular, an RL agent must consider the effects of its actions upon future rewards and observations: The exploration-exploitation tradeoff. In all but the most simple settings, the resulting inference is computationally intractable so that practical RL algorithms must resort to approximation. We demonstrate that the popular 'RL as inference' approximation can perform poorly in even very basic problems. However, we show that with a small modification the framework does yield algorithms that can provably perform well, and we show that the resulting algorithm is equivalent to the recently proposed K-learning, which we further connect with Thompson sampling.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Action and Perception as Divergence Minimization (2020, background): “The information gain term may also address concerns about maximum entropy RL raised by O’Donoghue et al. (2020).”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, background): “Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [60, 61, 34, 11, 57, 45], but many recently-proposed algorithms [2, 22, 48] are derived separately and their exact relationships are difficult to get out directly, due to mixing of algorithmic and ”
On the Global Convergence Rates of Softmax Policy Gradient Methods (2020, background): “While it is definitely true that entropy regularization encourages exploration, the form of exploration it encourages is not sensitive to epistemic uncertainty and as such it fails to provide a satisfactory solution to the exploration problem (e.g., O’Donoghue et al., 2020).”
Stochastic matrix games with bandit feedback (2020, background): “However, it also can be interpreted as a variational approximation to Thompson sampling [30] which incorporates deterministic optimism while maintaining many of the beneﬁts of Thompson sampling over UCB style approaches [32, 18].”