Multi-Task Reinforcement Learning with Soft Modularization

By Ruihan Yang, Huazhe Xu, Yi Wu, Xiaolong Wang. 2020

241 citations recorded by the source corpus.

Multi-task learning is a very challenging problem in reinforcement learning. While training multiple tasks jointly allow the policies to share parameters across different tasks, the optimization problem becomes non-trivial: It is unclear what parameters in the network should be reused across tasks, and the gradients from different tasks may interfere with each other. Thus, instead of naively sharing parameters across tasks, we introduce an explicit modularization technique on policy representation to alleviate this optimization issue. Given a base policy network, we design a routing network which estimates different routing strategies to reconfigure the base network for each task. Instead of creating a concrete route for each task, our task-specific policy is represented by a soft combination of all possible routes. We name this approach soft modularization. We experiment with multiple robotics manipulation tasks in simulation and show our method improves sample efficiency and performance over baselines by a large margin.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “More recent work (Devin et al., 2017; Frans et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2017; Meyerson and Miikkulainen, 2017; Ramachandran and Le, 2018; Chang et al., 2018; Liang et al., 2018; Alet et al., 2018; Cases et al., 2019; Yang et al., 2020) has focused on training neural networ”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Recent work focuses on learning a single multi-task policy with both onpolicy [34] and off-policy RL methods [35, 36, 37], and have shown promising results on a variety of robot manipulation tasks in simulation [34].”
Multi-Task Reinforcement Learning with Context-based Representations (2021, methodology): “…row) provides comparable performance to the other baselines, using it with a mixture PCGrad (Yu et al., 2020a) * 0.05 ± 0.076 Soft Modularization (Yang et al., 2020) 0.1 ± 0.089 SAC + FiLM (Perez et al., 2018) 0.2 ± 0.073 SAC + Metadata + ME ( CARE ) 0.3 ± 0.077 of encoders leads to even…”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “Hence, some recent work in multi-task RL focused on a trade-off between sharing and individualizing function modules [52], [53], [54].”
Conflict-Averse Gradient Descent for Multi-task Learning (2021, methodology): “By sharing parameters across tasks, MTL methods learn more efficiently with an overall smaller model size compared to learning with separate models [38, 40, 25].”
Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, background): “Some works also consider a fixed set of tasks [20], which are typically referred to as multi-task learning [33].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, methodology): “Despite the progress in multi-task learning, much work still centers around defining tasks as an orthonormal space (i.e., tasks are “one-hot” encoded in an orthonormal set [1], [2], [3]).”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “Finally, we note how many transfer and continual learning methods make use of weight freezing via masking to prevent catastrophic forgetting (Fernando et al., 2017; Mallya & Lazebnik, 2018; Golkar et al., 2019; Yang et al., 2020).”
Modular Lifelong Reinforcement Learning via Neural Composition (2022, background): “In RL, a few recent works have considered similar architectures, but without a substantial effort to study their applicability to truly compositional problems (Goyal et al., 2021; Yang et al., 2020).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Another closely related method also automatically learns a mapping from inputs to modular structures in the MTL setting, with applications to noncompositional robotic manipulation (Yang et al., 2020).”