Towards Generalization and Simplicity in Continuous Control

By Aravind Rajeswaran, Kendall Lowrey, E. Todorov, Sham M. Kakade. 2017

294 citations recorded by the source corpus.

This work shows that policies with simple linear and RBF parameterizations can be trained to solve a variety of continuous control tasks, including the OpenAI gym benchmarks. The performance of these trained policies are competitive with state of the art results, obtained with more elaborate parameterizations such as fully connected neural networks. Furthermore, existing training and testing scenarios are shown to be very limited and prone to over-fitting, thus giving rise to only trajectory-centric policies. Training with a diverse initial state distribution is shown to produce more global policies with better generalization. This allows for interactive control scenarios where the system recovers from large on-line perturbations; as shown in the supplementary video.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, background): “In the literature, similar lines of investigation have shown the differences in performance when comparing linear approximators, RBFs, and neural networks (Rajeswaran et al. 2017).”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “…were used in experiments in the generalization for RL literature, which have played an important role in advancing the ﬁeld (Packer et al., 2018; Rajeswaran et al., 2017; Pinto et al., 2017; Yu et al., 2017; Henderson et al., 2017a; Dulac-Arnold et al., 2020; Chevalier-Boisvert et al., 2018).”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, neutral): “[28], which at each iteration optimizes maxθ g> θj−1(θ − θj−1) subject to the quadratic constraint ‖θ − θj−1‖2Fθj−1 ≤ δ, where Fθ = E [ ∇θ log πθ(x,u)∇θ log πθ(x,u) ] is the approximate Fisher information of πθ [20].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “A number of algorithms based on MPC [23, 64], search-based planning [65, 25], dynamic programming [49, 26], or policy optimization [27, 51, 66, 67] can be used to approximately realize this.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, neutral): “Our work is similar in spirit to Rajeswaran et al. [47], which attempts to demystify the complexity of on-policy deep RL for continuous control.”
Boosting the Actor with Dual Critic (2017, methodology): “The RBF kernel bandwidth is chosen via median trick (Dai et al., 2014; Rajeswaran et al., 2017).”
Smoothed Dual Embedding Control (2017, methodology): “Although these operations are similar to Pwj−1π , we emphasize that the estimation of advantage, denoted as A(s, a), and the update of policy π are separated in NPG and TRPO.”
Towards Characterizing Divergence in Deep Q-Learning (2019, background): “(See, eg, (Lillicrap et al., 2016; Fujimoto et al., 2018b; Haarnoja et al., 2018b; Rajeswaran et al., 2017) for examples where networks in this range have previously been used.)”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “A number of pre-conditioned policy gradient methods have been developed in literature [19], [4], [35], [34], [43], [40], [44] and in principle any of them could be used.”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “For simplicity, we use the truncated natural policy gradient as described in [30].”
Simple random search provides a competitive approach to reinforcement learning (2018, methodology): “For the performance of these methods we used values reported by Rajeswaran et al. [27], Salimans et al. [28], Schulman et al. [31], and Haarnoja et al. [10].”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “Furthermore, their approach is designed as an exploration bonus for a single target task; in contrast, we define a new problem of efficiently optimizing a policy across a range of starting positions, which is considered a relevant to improve generalization in RL [19].”
Discrete Sequential Prediction of Continuous Actions for Deep RL (2017, methodology): “[34] Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade.”
Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (2018, methodology): “For policy optimization, we use a variant of the natural policy gradient method as described in Rajeswaran et al. (2017b).”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “The importance of having a wide and relevant state distribution for RL methods has been pointed out in numerous prior works [10, 43, 44].”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, background): “There has been some recent attempts in the RL community to study generalization to unseen initial conditions ( e.g. [45, 35]).”
Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms? (2018, background): “[17] Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M.”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “…RL context for three reasons: (1) In RL, data is streamed online and highly non-stationary, advocating for models capable of incremental FTL. (2) Many continuous control problems have a moderate number of dimensions that could fall within the capability of shallow models (Rajeswaran et al., 2017).”