Continual Model-Based Reinforcement Learning with Hypernetworks
By Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, Florian Shkurti. 2020
64 citations recorded by the source corpus.
Effective planning in model-based reinforcement learning (MBRL) and model-predictive control (MPC) relies on the accuracy of the learned dynamics model. In many instances of MBRL and MPC, this model is assumed to be stationary and is periodically re-trained from scratch on state transition experience collected from the beginning of environment interactions. This implies that the time required to train the dynamics model - and the pause required between plan executions - grows linearly with the size of the collected experience. We argue that this is too slow for lifelong robot learning and propose HyperCRL, a method that continually learns the encountered dynamics in a sequence of tasks using task-conditional hypernetworks. Our method has three main attributes: first, it includes dynamics learning sessions that do not revisit training data from previous tasks, so it only needs to store the most recent fixed-size portion of the state transition experience; second, it uses fixed-capacity hypernetworks to represent non-stationary and task-aware dynamics; third, it outperforms existing continual learning alternatives that rely on fixed-capacity networks, and does competitively with baselines that remember an ever increasing coreset of past experience. We show that HyperCRL is effective in continual model-based reinforcement learning in robot locomotion and manipulation scenarios, such as tasks involving pushing and door opening. Our project website with videos is at this link http://rvl.cs.toronto.edu/blog/2020/hypercrl/
Subfields: Artificial Intelligence, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Piotr Milo's
- Lukasz Kuci'nski
- Maciej Wolczyk
- Michal Zajkac
- Razvan Pascanu
- Yoshua Bengio
- Kyunghyun Cho
- Sergey Levine
In Control and Systems Engineering
Citations in the corpus
- Disentangling Transfer in Continual Reinforcement Learning (2022, background): “Although most of the research is concerned with model-free continual reinforcement learning, an approach to model-based continual RL was presented in [19].”