Learning Parameterized Skills

By Bruno Castro da Silva, George Dimitri Konidaris, Andrew Barto. 2012

215 citations recorded by the source corpus.

We introduce a method for constructing skills capable of solving tasks drawn from a distribution of parameterized reinforcement learning problems. The method draws example tasks from a distribution of interest and uses the corresponding learned policies to estimate the topology of the lower-dimensional piecewise-smooth manifold on which the skill policies lie. This manifold models how policy parameters change as task parameters vary. The method identifies the number of charts that compose the manifold and then applies non-linear regression in each chart to construct a parameterized skill by predicting policy parameters from task parameters. We evaluate our method on an underactuated simulated robotic arm tasked with learning to accurately throw darts at a parameterized target location.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Segment Anything (2023, background): “This capability is a form of task generalization [26].”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “…of the limits of what the robot can achieve in its task space; • Allows improving significantly the quality of learned inverse models in terms of speed of learning and generalization performance to reach goals in the task space, compared to different methods proposed in the literature;”
End-to-End Driving Via Conditional Imitation Learning (2017, methodology): “Parameterized goals have been used to train motion controllers in robotics [7], [8], [18].”
One-Shot Visual Imitation Learning via Meta-Learning (2017, background): “Another approach is to train controllers for a variety of tasks and learn a mapping from task representations to controller parameters [10, 25, 26].”
Hindsight Experience Replay (2017, background): “Learning simultaneously policies for multiple tasks have been heavily explored in the context of policy search, e.g. Schmidhuber and Huber (1990); Caruana (1998); Da Silva et al. (2012); Kober et al. (2012); Devin et al. (2016); Pinto and Gupta (2016).”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, background): “Thus, in the ﬁrst stage our architecture learns a parameterized skill (da Silva et al., 2012) to perform different subtasks depending on input parameters.”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, methodology): “Many existing methods [7, 33] employ an ensemble approach by learning a discrete set of control policies and consolidating them into one regression model.”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, background): “Critically, unlike the works on contextual policies (Caruana, 1998; Da Silva et al., 2012; Kober et al., 2012) which require on-policy trajectories with each new goal, the value function approaches such as Horde (Sutton et al., 2011) and UVF (Schaul et al., 2015) can reuse off-policy data to learn…”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, background): “We wish to learn a policy π∗θ(s) that performs well for allM∼ D. Note that this robust policy does not have an explicit dependence on p, and we require it to perform well without knowledge of p.”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “Both value functions and policies can be goal-conditional [Schmidhuber and Huber, 1991; Kaelbling, 1993; Da Silva et al., 2012; Kupcsik et al., 2013; Deisenroth et al., 2014; Schaul et al., 2015; Pong et al., 2018] and learned by relabeling experiences in hindsight [Kaelbling, 1993; Andrychowicz et…”
Learning to Learn: Meta-Critic Networks for Sample Efficient Learning (2017, background): “Parametrised Tasks: In learning for control, the contextual policy or parametrised task setting considers actors that are parametrised by some metadata describing the goal they should achieve (e.g., the goal state of a mobile agent or robot arm to reach) [7, 19, 29].”
Hindsight policy gradients (2017, background): “Recent examples of learning goal-conditional behavior include the works of Da Silva et al. [3], Schmidhuber [4], Srivastava et al. [5], Kupcsik et al. [6], Deisenroth et al. [7], Fabisch and Metzen [8], Schaul et al. [9], Zhu et al. [10], Held et al. [11].”
Probabilistic Movement Primitives (2013, methodology): “MPs have been used to solve many complex tasks, including ‘Ball-in-the-Cup’ [4], Ball-Throwing [5, 6], Pancake-Flipping [7] and Tetherball [8].”
Learning Robust Options (2018, background): “Temporally Extended Actions, also known as options (Sutton, Precup, and Singh 1999), skills (da Silva, Konidaris, and Barto 2012; Mankowitz, Mann, and Mannor 2016b; 2016a) or macro-actions (Hauskrecht et al. 1998) have been shown both theoretically (Precup, Sutton, and Singh 1998) and…”
Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control (2019, methodology): “We present a method that combines parameterized skill learning [4, 5] and end-to-end deep visuomotor skill learning from demonstration [18, 26].”