CoMPS: Continual Meta Policy Search

By Glen Berseth, Zhiwei Zhang, Grace Zhang, Chelsea Finn, Sergey Levine. 2021

20 citations recorded by the source corpus.

We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce a new method, continual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning a new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences of challenging continuous control tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Eric Eaton
2. Jorge Armando Mendez Mendez
3. Jun Zhu
4. Hang Su
5. Irina Rish
6. Matthew Riemer
7. Xingxing Zhang
8. Khimya Khetarpal

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Matthew Riemer
6. Eric Eaton
7. Irina Rish
8. Jorge Armando Mendez Mendez

In Reinforcement Learning

1. Jorge Armando Mendez Mendez
2. Matthew Riemer
3. Eric Eaton
4. Khimya Khetarpal
5. Irina Rish
6. Liyuan Wang
7. Doina Precup
8. Jun Zhu

In Meta & Continual Learning

1. Jorge Armando Mendez Mendez
2. Eric Eaton
3. Xingxing Zhang
4. Liyuan Wang
5. Hang Su
6. Irina Rish
7. Jun Zhu
8. Matthew Riemer

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “COMPS [35] considers continual meta-training of RL, using behavioral cloning to quickly learn a new task from the knowledge of previously-learned incremental tasks.”
• Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “There have been a number of recent improvements on this topic of meta-optimization in multi-task RL (Nichol and Schulman, 2018; Kim et al., 2018; Rothfuss et al., 2018; Flennerhag et al., 2018; Nagabandi et al., 2018; Mendonca et al., 2019; Finn et al., 2019; Lin et al., 2020; Berseth et al., 2021; ”
• Building a Subspace of Policies for Scalable Continual Learning (2022, unknown): “…of these challenges (Tessler et al., 2017; Parisi et al., 2019; Khetarpal et al., 2020; Powers et al., 2021; Wołczyk et al., 2021; Tessler et al., 2017; Berseth et al., 2022; Nagabandi et al., 2018; Xie et al., 2020; Xu et al., 2020; Berseth et al., 2021; Ren et al., 2022; Kessler et al., 2022).”
• How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Berseth et al. (2022) also used replay in longer sequences of RL tasks in conjunction with off-line meta-RL to achieve lifelong transfer.”