A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
By Nicholas C. Landolfi, Garrett Thomas, Tengyu Ma. 2019
19 citations recorded by the source corpus.
The aim of multi-task reinforcement learning is two-fold: (1) efficiently learn by training against multiple tasks and (2) quickly adapt, using limited samples, to a variety of new tasks. In this work, the tasks correspond to reward functions for environments with the same (or similar) dynamical models. We propose to learn a dynamical model during the training process and use this model to perform sample-efficient adaptation to new tasks at test time. We use significantly fewer samples by performing policy optimization only in a "virtual" environment whose transitions are given by our learned dynamical model. Our algorithm sequentially trains against several tasks. Upon encountering a new task, we first warm-up a policy on our learned dynamical model, which requires no new samples from the environment. We then adapt the dynamical model with samples from this policy in the real environment. We evaluate our approach on several continuous control benchmarks and demonstrate its efficacy over MAML, a state-of-the-art meta-learning algorithm, on these tasks.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Tengyu Ma
- Chelsea Finn
- Marcin Andrychowicz
- Yoshua Bengio
- Ruslan Salakhutdinov
- Peter Welinder
Citations in the corpus
- Solving Rubik's Cube with a Robot Hand (2019, background): “A common theme in research is to try to exploit a shared structure in a distribution of environments, to quickly identify and adapt to previously unseen cases [55, 94, 47, 54] There are works that directly treat meta-learning as identification of a dynamics model of the environment [19] while others”
- Model-based Adversarial Meta-Reinforcement Learning (2020, background): “, 2019), and are particularly suitable for meta-RL settings where the optimal policies for tasks are very different, but the underlying dynamics is shared (Landolfi et al., 2019).”