Randomized Value Functions via Multiplicative Normalizing Flows

By Ahmed Touati, Harsh Satija, Joshua Romoff, Joelle Pineau, Pascal Vincent. 2018

41 citations recorded by the source corpus.

Randomized value functions offer a promising approach towards the challenge of efficient exploration in complex environments with high dimensional state and action spaces. Unlike traditional point estimate methods, randomized value functions maintain a posterior distribution over action-space values. This prevents the agent's behavior policy from prematurely exploiting early estimates and falling into local optima. In this work, we leverage recent advances in variational Bayesian neural networks and combine these with traditional Deep Q-Networks (DQN) and Deep Deterministic Policy Gradient (DDPG) to achieve randomized value functions for high-dimensional domains. In particular, we augment DQN and DDPG with multiplicative normalizing flows in order to track a rich approximate posterior distribution over the parameters of the value function. This allows the agent to perform approximate Thompson sampling in a computationally efficient manner via stochastic gradient methods. We demonstrate the benefits of our approach through an empirical comparison in high dimensional environments.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Other methods have examined approximating the Bayesian posterior of the value function (Osband et al., 2016; 2018; Touati et al., 2018), again using the variance to encourage exploration to unseen regions of the state space.”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “However, despite this clear failing (2) is precisely the loss used by the majority of approaches to VI for RL [17, 35, 65, 69, 20].”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “Another series of methods [52, 32, 33] aim to estimate uncertainty to determine the trustworthiness of a Q-value prediction.”
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning (2021, methodology): “(2018), random prior based methods including dropout have been widely applied to capture uncertainty in RL (Gal & Ghahramani, 2016a; Osband et al., 2018; Fortunato et al., 2017; Lipton et al., 2018; Tang & Kucukelbir, 2017; Touati et al., 2020).”
Normalizing Flows: An Introduction and Review of Current Methods (2020, background): “ca [Mazoure et al., 2019; Nadeem Ward et al., 2019; Touati et al., 2019], computer graphics [Müller et al.”
TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed (2022, background): “Applications include image generation [15,17], noise modelling [18], video generation [19], audio generation [20–22], graph generation [23], reinforcement learning [24–26], computer graphics [67], and physics [27–31].”