VIREL: A Variational Inference Framework for Reinforcement Learning

By Matthew R. Fellows, Anuj Mahajan, Tim G. J. Rudner, Shimon Whiteson. 2018

59 citations recorded by the source corpus.

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in pseudo-likelihood methods and difficulties learning deterministic policies in maximum entropy RL based approaches. We propose VIREL, a novel, theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP. This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods from VIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Making Sense of Reinforcement Learning and Probabilistic Inference (2020, methodology): “It is possible to view the algorithms of the ‘RL as inference’ approach in this light (Rawlik et al., 2013; Todorov, 2009; Toussaint, 2009; Deisenroth et al., 2013; Fellows et al., 2019); see Levine (2018) for a recent survey.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, background): “Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [60, 61, 34, 11, 57, 45], but many recently-proposed algorithms [2, 22, 48] are derived separately and their exact relationships are difficult to get out directly, due to mixing of algorithmic and ”
HyQE: Ranking Contexts with Hypothetical Query Embeddings (2024, background): “It has been extensively studied across many fields of machine learning (Kingma and Welling, 2013; Hoffman et al., 2013; Zhou and Li, 2022; Fellows et al., 2019).”