Actor-Critic Reinforcement Learning with Energy-Based Policies

By Nicolas Manfred Otto Heess, David Silver, Yee Whye Teh. 2012

73 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “Deep neural networks have been used to estimate the environment E ; restricted Boltzmann machines have been used to estimate the value function [21]; or the policy [9].”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “A closely related idea is presented in the work on energy-based reinforcement learning (Sallans and Hinton, 2004; Heess et al., 2013), where the free energy of an energy-based model (in that case, a restricted Boltzmann machine) is adjusted based on a reinforcement learning update rule, such that…”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “A number of prior works have also explored representing policies as energy-based models, with the Q-value obtained from an energy model such as a restricted Boltzmann machine (RBM) (Sallans & Hinton, 2004; Elfwing et al., 2010; Otsuka et al., 2010; Heess et al., 2012).”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “In Heess et al. (2012) this was extended to use an actor-critic algorithm instead of TD-learning, however the two updates were not combined as we have done in this paper.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “1, except that C is now an FNN or RNN trained by evolutionary algorithms [200, 255, 105, 56, 68] applied to NNs [165, 321, 180, 259, 72, 90, 89, 110, 94], or by policy gradient methods [314, 315, 316, 274, 18, 1, 63, 128, 313, 210, 192, 191, 256, 85, 312, 190, 82, 93][245, Sec.”
Deterministic Policy Gradient Algorithms (2014, background): “However, because the deterministic policy gradient removes the integral over actions, we can avoid importance sampling in the actor; and by using Q-learning, we can avoid importance sampling in the critic.”
Energy-Based Imitation Learning (2020, methodology): “Thanks to the appealing features, EBMs have been introduced into many RL literature, for instance, parameterized as value function [36], employed in the actor-critic framework [21], applied to MaxEnt RL [19] and modelbased RL regularization [3].”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, background): “Efforts to combine reinforcement learning (RL) and probabilistic inference have a long history, spanning diverse fields such as control, robotics, and RL [64, 62, 46, 47, 27, 74, 75, 73, 36].”