Learning Complex Neural Network Policies with Trajectory Optimization

By Sergey Levine, Vladlen Koltun. 2014

146 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

End-to-End Training of Deep Visuomotor Policies (2015, background): “…set for this supervised learning procedure can be constructed from example demonstrations [20], trajectory optimization under known dynamics [21, 22, 28], and trajectory-centric reinforcement learning methods that operate under unknown dynamics [19, 23], which is the approach taken in this work.”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “…(Levine and Koltun, 2013a; Nachum et al., 2017a,b), model-based algorithms based on the guided policy search framework (Levine and Abbeel, 2014; Levine and Koltun, 2014; Levine et al., 2016), and a variety of methods based on soft Q-learning (Haarnoja et al., 2017; Schulman et al., 2017) and…”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “[24] applied guided policy search to learn bipedal locomotion in simulation.”
Distral: Robust multitask reinforcement learning (2017, background): “Another related line of work is guided policy search [17, 18, 15, 16].”
Learning contact-rich manipulation skills with guided policy search (2015, methodology): “The policy KL-divergence term in the trajectory objective requires a slightly different dynamic programming pass, which is described in prior work [21], [1].”
3D Simulation for Robot Arm Control with Deep Q-Learning (2016, neutral): “This decomposes the problem into a trajectory optimization phase, which generates training data for a supervised learning phase to train a policy [4].”
Learning agile and dynamic motor skills for legged robots (2019, background): “Levine and Koltun [21] combined learning and trajectory optimization to train a locomotion controller for a simulated 2D walker.”
Terrain-adaptive locomotion skills using deep reinforcement learning (2016, methodology): “…of action abstraction as compared to other recent deep-RL methods that attempt to learn control policies that are directly based on joint torques, e.g., [Levine and Abbeel 2014; Levine and Koltun 2014; Mordatch and Todorov 2014; Mordatch et al. 2015; Heess et al. 2015; Lillicrap et al. 2015].”
Interactive Control of Diverse Complex Characters with Neural Networks (2015, methodology): “While previous work [9, 11] used ADMM or dual gradient descent to solve similar optimization problems, it is non-trivial to adapt them to asynchronous and stochastic setting we have.”
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics (2014, methodology): “Previous guided policy search methods used model-based trajectory optimization algorithms that required known, differentiable system dynamics [12, 13, 14].”
Learning symmetric and low-energy locomotion (2018, methodology): “Two major classes of policy search methods used in the previous work include sampling-based methods [Geijtenbeek et al. 2013; Tan et al. 2014] and gradient-based methods [Levine and Koltun 2014; Liu et al. 2016; Peng et al. 2016, 2017].”
Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search (2015, methodology): “Guided policy search has been applied to locomotion [21], robotic manipulation [22], and visionbased robotic control [23], but all prior applications rely on an offline trajectory optimization phase to generate the controller that is then executed on the real system.”
DeepLoco (2017, methodology): “Numerous methods have been applied to tasks that include planar biped locomotion or planar swimming: trajectory optimization [Levine and Abbeel 2014; Levine and Koltun 2014; Mordatch and Todorov 2014]; trust region policy optimization (TRPO) [Schulman et al. 2015]; and actor-critic approaches…”
Learning deep neural network policies with continuous memory states (2015, methodology): “Together, the cost and the penalty form the following objective: The linear-Gaussian controllers p ( u t | x t ) can be optimized in a variety of ways, including ofﬂine trajectory optimization methods with known models [18] and trajectory-centric reinforcement learning [9].”
Combining the benefits of function approximation and trajectory optimization (2014, result): “As noted earlier, the recent approaches of [13] and concurrent [15] are related to ours.”
Combining model-based policy search with online model learning for control of physical humanoids (2016, methodology): “We instead follow an approach of learning control policies based on simulation, following recent approaches [14], [16], [23].”
Approximate real-time optimal control based on sparse Gaussian process models (2014, unknown): “Similarly, model-based policy search methods can search the policy space offline using a system model – hence minimizing the interactions with the real system – and discover successful policies on complex dynamical systems in as few as 10 episodes [7], [8].”