Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

By Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, Sergey Levine. 2019

832 citations recorded by the source corpus.

Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this paper, we address these challenges by developing an off-policy meta-RL algorithm that disentangles task inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to integrate these task variables with off-policy RL algorithms to achieve both meta-training and adaptation efficiency. Our method outperforms prior algorithms in sample efficiency by 20-100X as well as in asymptotic performance on several meta-RL benchmarks.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Chelsea Finn
  3. Yoshua Bengio
  4. Pieter Abbeel
  5. Ruslan Salakhutdinov
  6. Mohit Bansal
  7. Dacheng Tao
  8. Quoc V. Le

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Abhishek Gupta
  5. Shimon Whiteson
  6. Ofir Nachum
  7. Shixiang Shane Gu
  8. George Tucker

In Meta & Continual Learning

  1. Chelsea Finn
  2. Sergey Levine
  3. Han-Jia Ye
  4. Dacheng Tao
  5. James Harrison
  6. Annie Xie
  7. Abhishek Gupta
  8. Pieter Abbeel

Citations in the corpus