Variational Policy Search via Trajectory Optimization

By Sergey Levine, Vladlen Koltun. 2013

133 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, methodology): “…set for this supervised learning procedure can be constructed from example demonstrations [20], trajectory optimization under known dynamics [21, 22, 28], and trajectory-centric reinforcement learning methods that operate under unknown dynamics [19, 23], which is the approach taken in this…”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “…into this category (Neumann, 2011; Levine and Koltun, 2013b), sometimes with the modification of using explicit trajectory optimization rather than reweighting to construct the target actions (Levine and Koltun, 2013b), and sometimes using an exponential transformation on r(τ) to ensure positivity.”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, methodology): “When combined with receding horizon control [3], and machine learning methods for learning approximate system models, such algorithms are a powerful tool for solving complicated control problems [3, 4, 5, 6]; however, they either rely on a known system model or require the design of relatively low-d”
Maximum a Posteriori Policy Optimisation (2018, methodology): “Existing EM policy search approaches perform this step typically by reweighting trajectories with sample returns (Kober & Peters, 2009) or via local trajectory optimization (Levine & Koltun, 2013).”
Distral: Robust multitask reinforcement learning (2017, background): “Another related line of work is guided policy search [17, 18, 15, 16].”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “Some work in robotics has considered exploiting supervision as a means to provide indirect sampling guidance to improve policy search methods that maximize task reward [18, 19, 26], but these approaches do not make use of maximum likelihood training.”
Learning contact-rich manipulation skills with guided policy search (2015, methodology): “Previous guided policy search algorithms have also focused on leveraging known models of the environment [19], [20], [21] and more sophisticated trajectory optimization algorithms [22].”
Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, positive): “…given the observations, P ( s t | x ≤ t ) , we represent the current value as a random variable, We follow the idea of RL as probabilistic inference ( 39, 71, 72 ) which leads to a Kullback-Leibler divergence (KL) regularised objective in which the policy Q is regularised against a prior policy P .”
PLATO: Policy learning using adaptive trajectory optimization (2016, critical): “This assumption is not always realistic [17].”
Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations (2016, background): “They also proposed forcing the supervisor to guide the robots policy to better regions of the state space [15].”
Neural networks and differential dynamic programming for reinforcement learning problems (2016, unknown): “We consider some assumptions to simplify (A).”
Combining the benefits of function approximation and trajectory optimization (2014, methodology): “As noted earlier, the recent approaches of [13] and concurrent [15] are related to ours.”
Learning Complex Neural Network Policies with Trajectory Optimization (2014, methodology): “The initial trajectory for the swimmer was generated with unconstrained trajectory optimization using DDP, while the initial walking trajectory used a demonstration from a hand-crafted locomotion system (Yin et al., 2007), following our prior work (Levine & Koltun, 2013a).”
Approximate real-time optimal control based on sparse Gaussian process models (2014, unknown): “Similarly, model-based policy search methods can search the policy space offline using a system model – hence minimizing the interactions with the real system – and discover successful policies on complex dynamical systems in as few as 10 episodes [7], [8].”
Guided Policy Search as Approximate Mirror Descent (2016, methodology): “In fact, a variational formulation of guided policy search [7] corresponds to the EM algorithm, where the C-step is actually the E-step, and the S-step is the M-step.”
Exploring Deep and Recurrent Architectures for Optimal Control (2013, methodology): “A more recent variational variant of guided policy search eschews importance sampling and allows for any supervised learning algorithm, including SGD, to be used [16].”
Robust Locally-Linear Controllable Embedding (2017, background): “Algorithms from this class such as differential dynamic programming (DDP) [3], iterative linear quadratic regulator (iLQR) [8], and iterative linear quadratic Gaussian (iLQG) [14] have been successfully applied to a variety of complex control problems [1, 13, 7, 9].”
Recall Traces: Backtracking Models for Efficient Reinforcement Learning (2018, background): “Variational inference has also been proposed for policy search (Neumann et al., 2011; Levine & Koltun, 2013).”