Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

By Sergey Levine. 2018

873 citations recorded by the source corpus.

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “We can draw connections to the control as inference framework [20] for the constrained RL problem outlined in 3.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “The connections between RL and divergence minimization have also been explored, with studies showing that entropy regularization in RL can be viewed as minimizing reverse KL divergence between reward-weighted trajectory and policy trajectory distributions (Kappen et al., 2013; Levine, 2018).”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Both KL-control and MaxEnt RL can be derived from a general framework of control-as-inference (Levine, 2018) which poses control as minimising KL from a certain target distribution.”
Language Model Cascades (2022, background): “This can be cast as a reinforcement learning problem with string-valued actions, or equivalently as an inference problem where we condition on the goal state that A T = ‘yes’ for the soonest possible T (c.f., planning as inference (Levine, 2018)).”
An Introduction to Probabilistic Programming (2018, background): “A large number of computational neuroscientists, cognitive scientists, machine learning researchers believe in a well-established formal model of how agents choose and plan their actions (Levine, 2018; Tenenbaum et al., 2011; Ghahramani, 2015).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “Even value functions can be viewed as performing inference in a graphical model with auxiliary optimality variables, amounting to estimation of the distribution over future rewards (Levine, 2018).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…= r(s,a) − αf(π(a|s)/πβ(a|s)), with the important special case of the KL-divergence corresponding to r̄(s,a) = r(s,a) + α log πβ(a|s) − α log π(a|s), with the last term subsumed inside the entropy regularizer H(π(·|s)) when using a maximum entropy reinforcement learning algorithm (Levine, 2018).”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “Levine (2018) reformulated RL and control problems using probabilistic inference, which allows us to bring to bear a large pool of approximate inference methods, and flexibly extend the models.”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, methodology): “[17] use a learned prior over actions in the maximum entropy RL framework [26] and justify it as a regularizer based on mutual information.”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, methodology): “Given costs c ( s i | φ ) , this can be treated by including an optimality variable C in G , where p ( C = 1 | s 1: T , φ ) ∝ =1 exp − c ( s t | φ ) (Todorov, 2007; Levine, 2018).”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “These approaches are very closely related to maximum entropy reinforcement learning techniques which add an entropy regularization ‘bonus’ to prevent early convergence to deterministic policies and thereby heuristically encourage exploration [57, 59, 28, 35, 2, 25].”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “Substituting the KL divergence with the entropy recovers the ELBO derived in [18].”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “In the literature it is often formulated as under the guise of RL as inference (Levine, 2018) and is solved using the classical expectationmaximization (EM) algorithm.”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “As discussed in prior work [13], this has an appealing theoretical interpretation based on a connection to control as inference [24], and in practice can provide some degree of shaping, as log-probabilities often increase smoothly as the agent approaches the goal.”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “There are also conceptual connections to ideas formulated around “RL as inference”, where the solution approach is to infer the values of actions conditioned on the trajectory being optimal [Attias, 2003; Toussaint and Storkey, 2006] (see [Levine, 2018] for a review).”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, background): “Speciﬁcally, they have shown that optimal control under entropy regularization can be viewed as approximate inference on a graphical model, or equivalently minimizing reverse KL divergence between reward-weighted trajectory and policy trajectory distributions [20, 24].”
RL with KL penalties is better viewed as Bayesian inference (2022, methodology): “The Bayesian perspective moves KL-regularised RL closer to other divergence-minimisation-based approaches to ﬁne-tuning LMs (Khalifa et al., 2021) and, more broadly, to other divergence-minimisation-based accounts of control (Levine, 2018; Hafner et al., 2020).”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, background): “An alternative approach to ﬁnding optimal policies derives from research that treats RL as a problem in probabilistic inference, including Maximum a Posteriori Policy Optimization (MPO) (Levine, 2018; Abdolmaleki et al., 2018a;b).”