Variational Inference for Policy Search in changing situations

By Gerhard Neumann. 2011

116 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “This approach has lead to a variety of EM-based RL algorithms (Peters et al., 2010; Neumann, 2011; Abdolmaleki et al., 2018), an early example of which is reward-weighted regression (RWR) (Peters & Schaal, 2007).”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “…vector θ. Assuming r(τ) is positive and bounded and applying Jensen’s inequality results in a variety of algorithms (Peters and Schaal, 2007; Hachiya et al., 2009; Neumann, 2011; Abdolmaleki et al., 2018), including reward-weighted regression (Peters and Schaal, 2007), that all follow the…”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “A number of related methods have also used maximum entropy policy optimization as an intermediate step for optimizing policies under a standard expected reward objective (Pe-Reinforcement ters et al., 2010; Neumann, 2011; Rawlik et al., 2012; Fox et al., 2016).”
Maximum a Posteriori Policy Optimisation (2018, methodology): “A number of other authors have considered the same perspective but in a model-free RL setting (Neumann, 2011; Peters et al., 2010a; Florensa et al., 2017; Daniel et al., 2016) or inverse RL problems (Ziebart et al., 2008).”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, background): “A number of prior works have explored how reinforcement learning can be cast in the framework of probabilistic inference (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008; Toussaint, 2009; Peters et al., 2010; Neumann, 2011).”
Lifted Stochastic Planning, Belief Propagation and Marginal MAP (2018, methodology): “Solutions include the expectation maximization (EM) or variational EM algorithms (Toussaint and Storsky 2006; Furmston and Bar-ber 2010; Neumann 2011; Kober and Peters 2011) policy gradients (Kober and Peters 2011; van de Meent et al. 2016) and BP (Liu and Ihler 2012; Cheng et al. 2013).”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “We center our analyses around MPO, AWR, and SAC, because they are representative algorithms that span both EM-based [49, 50, 43, 44, 1, 56, 42] and KL-control-based [59, 51, 12, 29, 21, 33, 32] RL and achieve some of the most competitive performances on popular benchmarks [6, 58].”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, methodology): “Methods that derive from the pseudo-likelihood inference framework [12, 64, 46, 26, 44, 1] and use expectationmaximisation (EM) favour risk-seeking policies [34], which can be suboptimal.”
Learning concurrent motor skills in versatile solution spaces (2012, background): “Additionally, as Neumann [12] has shown, representing the policy by a single Gaussian can affect the performance of policy search methods due to averaging over several modes.”
Model-based contextual policy search for data-efficient generalization of robot skills (2017, methodology): “In contextual policy search [21, 27], our goal is to generalize the lower-level control policy u = π(x;ω) to multiple contexts.”
Deriving and improving CMA-ES with information geometric trust regions (2017, methodology): “Similar to prior work [15], we can now introduce a variational distribution q(x ) which is used to decompose the logp (R;θ ), i.”
Information-Theoretic Motor Skill Learning (2013, background): “Contextual policy search (Kober, Oztop, and Peters 2010; Neumann 2011) aims to adapt the parameters ω of a lower-level policy to the current context s .”
A Bayesian Approach to Generative Adversarial Imitation Learning (2018, background): “…the same insight behind the probabilistic inference formulation of reinforcement learning, in which the reinforcement learning problem is casted into the probabilistic inference problem by introducing the auxiliary return optimality event [Toussaint, 2009, Neumann, 2011, Abdolmaleki et al., 2018].”
Hierarchical Relative Entropy Policy Search (2014, methodology): “We treat the hierarchical structure of the policy as a latent variable, which results in the Hierarchical Relative Entropy Policy Search (HiREPS) algorithm.”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “MaxEnt RL has appealing connections to probabilistic inference [8, 23, 34, 40, 49, 52, 54, 61], prompting a renewed interest in recent years [1, 17, 27].”
Data-Efficient Generalization of Robot Skills with Contextual Policy Search (2013, methodology): “Model-free methods have been applied to contextual policy search (Kober et al. 2010; Neumann 2011).”
Black-Box Policy Search with Probabilistic Programs (2015, methodology): “To do so we employ a planning as inference interpretation [Toussaint et al., 2006, Rawlik et al., 2012, Neumann, 2011, Hoffman et al., 2009a,b, Levine and Koltun, 2013] that casts policy search as stochastic gradient ascent on the marginal likelihood.”