Reinforcement Learning with Deep Energy-Based Policies

By Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine. 2017

1,623 citations recorded by the source corpus.

We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before. We apply our method to learning maximum entropy policies, resulting into a new algorithm, called soft Q-learning, that expresses the optimal policy via a Boltzmann distribution. We use the recently proposed amortized Stein variational gradient descent to learn a stochastic sampling network that approximates samples from this distribution. The benefits of the proposed algorithm include improved exploration and compositionality that allows transferring skills between tasks, which we confirm in simulated experiments with swimming and walking robots. We also draw a connection to actor-critic methods, which can be viewed performing approximate inference on the corresponding energy-based model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Generative Models

Citations in the corpus

Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “Haarnoja et al. [2017]. In practice, we treat τ as a hyperparameter.”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “π∗ Ψ(at|st) = exp(Ψ(st, at)) (9) Because it avoids taking a hard max over noisy estimates, Ψ-learning leads to less overestimation of future reward [1, 21].”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “, 2005), maximum entropy RL (Haarnoja et al., 2017) or KL-control (Todorov, 2007).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “To test our algorithm against a state-of-the-art of-ﬂine deep RL technique, we implement a discrete version of Batch Constrained Q-learning (Fujimoto et al., 2018), DBCQ.”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In another class of methods (Haarnoja et al., 2017, 2018a), the latent variable z is selected independently at each time step, and the policy π(a|s, z) has some simple unimodal form (e.g., a Gaussian distribution) conditioned on z, but a complex multimodal form when z is integrated out.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “Maximum entropy reinforcement learning alters the RL objective, though the original objective can be recovered using a temperature parameter (Haarnoja et al., 2017).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Indeed, other works which focus on exploration in continuous control have found the need to propose exploration-advanageous variants of these standard benchmarks (Haarnoja et al., 2017; Houthooft et al., 2016).”
Equivalence Between Policy Gradients and Soft Q-Learning (2017, methodology): “• Haarnoja et al. [2017] work in the same setting of soft Q-learning as the current paper, and they are concerned with tasks with high-dimensional action spaces, where we would like to learn stochastic policies that are multi-modal, and we would like to use Q-functions for which there is no closed-f”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Even though entropy regularized policy optimization is a well studied topic in RL [46, 39, 40, 47, 5, 4, 6, 7]–in fact, one that has been attracting renewed interest from concurrent work [25, 11]–we contribute new observations to this study that are essential for the methods we propose: first, we id”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…a number of different variants of actor-critic methods, including on-policy variants that directly estimate V π(s) (Konda and Tsitsiklis, 2000), and off-policy variants that estimate Qπ(s,a) via a parameterized state-action value function Qπφ(s,a) (Haarnoja et al., 2018, 2017; Heess et al., 2015).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “We compare to: prior offline RL methods that use a policy constraint – BEAR [30] and BRAC [59]; SAC [19], an off-policy actor-critic method that we adapt to offline setting; and behavioral cloning (BC).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Two methods with common authors are energy-based policies [30] and more recent and with widespread use, the Soft Actor Critic (SAC) algorithm [31], both rely on adding an entropy term to the reward function, so we update the policy objective from Eqn.”
Self-Imitation Learning (2018, methodology): “The goal of entropy-regularized RL is to learn a stochastic policy which maximizes the entropy of the policy as well as the γ -discounted sum of rewards (Haarnoja et al., 2017; Ziebart et al., 2008): where is the entropy of the policy π , and α ≥ 0 represents the weight of entropy bonus.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “, 2015) and actor-critic algorithms (Haarnoja et al., 2017; Konda and Tsitsiklis, 2002), is a pathological interaction between the data distribution induced by the latest policy, and the errors induced in the learned approximate value function as a consequence (1)UC Berkeley.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “…2017; Geist et al., 2019) has been investigated for many diﬀerent purposes in the literature, including exploration (de Farias and Van Roy, 2000; Haarnoja et al., 2017, 2018), smoothing (Dai et al., 2018b), avoiding premature convergence (Nachum et al., 2017a), ensuring tractability (Todorov,…”
Generalized Off-Policy Actor-Critic (2019, background): “This line of work includes the deep energy-based RL (Haarnoja et al., 2017, 2018), where a value function is learned off-policy and the policy is derived from the value function directly, and path consistency learning (Nachum et al., 2017a,b), where gradients are computed to satisfy certain path…”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “, 2015) and SAC (Haarnoja et al., 2017) to improve the stability of learning.”
Boosting the Actor with Dual Critic (2017, methodology): “Their algorithm is based on the duality of entropy-regularized Bellman equation (Todorov, 2007; Rubin et al., 2012; Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), rather than the exact Bellman optimality equation used in our work.”
Smoothed Dual Embedding Control (2017, methodology): “Different from the motivation in our method where the entropy regularization is introduced in dual form for smoothing (Nesterov, 2005), the entropy-regularized MDP has been proposed for balancing exploration and exploitation (Haarnoja et al., 2017), taming the noises in observations (Rubin et al.”