Multitask Soft Option Learning
By Maximilian Igl, Andrew Gambardella, Nantas Nardelli, N. Siddharth, Wendelin Böhmer, Shimon Whiteson. 2019
26 citations recorded by the source corpus.
We present Multitask Soft Option Learning(MSOL), a hierarchical multitask framework based on Planning as Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This ''soft'' version of options avoids several instabilities during training in a multitask setting, and provides a natural way to learn both intra-option policies and their terminations. Furthermore, it allows fine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hierarchical policy. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines.
Subfields: Artificial Intelligence, Finance, Ocean Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Joelle Pineau
- Chelsea Finn
- Pieter Abbeel
- Qiang Yang
- Nicolas Manfred Otto Heess
- Yoshua Bengio
- Ruslan Salakhutdinov
In Finance
- Joelle Pineau
- Jost Tobias Springenberg
- Abbas Abdolmaleki
- Nicolas Manfred Otto Heess
- Martin A. Riedmiller
In Ocean Engineering
Citations in the corpus
- A Survey on Multi-Task Learning (2017, background): “In [150], multi-task soft option learning, a hierarchical framework based on planning as inference, is regularized by a shared”
- Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, methodology): “…models have been used for the discovery of behavior abstractions as part of an end-to-end reinforcement learning paradigm [e.g. Teh et al., 2017, Igl et al., 2019, Tirumala et al., 2019, Galashov et al., 2018] where the models act as learned inductive biases that induce the sharing of behavior…”
- Block Contextual MDPs for Continual Learning (2021, background): “Several works proposed a mixture-of-experts based approach to adaptation where an expert model is learned for each task (Doya et al., 2002; Neumann et al., 2009; Peng et al., 2016) or experts are shared across tasks (Igl et al., 2020; Sodhani et al., 2021b).”
- Multi-Task Reinforcement Learning with Context-based Representations (2021, background): “Multi-task reinforcement learning (MTRL) is a promising approach to train effective real-world agents (Tanaka & Yamamura, 2003; Rusu et al., 2016; Borsa et al., 2016; Rajeswaran et al., 2016; El Bsat et al., 2017; Andreas et al., 2017; Igl et al., 2020; D’Eramo et al., 2020; Yu et al., 2020a).”