MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies

By Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, Sergey Levine. 2019

230 citations recorded by the source corpus.

Humans are able to perform a myriad of sophisticated tasks by drawing upon skills acquired through prior experience. For autonomous agents to have this capability, they must be able to extract reusable skills from past experience that can be recombined in new ways for subsequent tasks. Furthermore, when controlling complex high-dimensional morphologies, such as humanoid bodies, tasks often require coordination of multiple skills simultaneously. Learning discrete primitives for every combination of skills quickly becomes prohibitive. Composable primitives that can be recombined to create a large variety of behaviors can be more suitable for modeling this combinatorial explosion. In this work, we propose multiplicative compositional policies (MCP), a method for learning reusable motor skills that can be composed to produce a range of complex behaviors. Our method factorizes an agent's skills into a collection of primitives, where multiple primitives can be activated simultaneously via multiplicative composition. This flexibility allows the primitives to be transferred and recombined to elicit new behaviors as necessary for novel tasks. We demonstrate that MCP is able to extract composable skills for highly complex simulated characters from pre-training tasks, such as motion imitation, and then reuse these skills to solve challenging continuous control tasks, such as dribbling a soccer ball to a goal, and picking up an object and transporting it to a target location.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “While some works (Bacon et al., 2017; Stolle & Precup, 2002; Peng et al., 2019) learn a discrete set of lower-level policies, each behaving as a primitive skill, other works (Vezhn-evets et al., 2017; Nachum et al., 2018b; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills…”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, methodology): “Comparing R4 and R5 shows that PMCP is effective in adding fail-state recovery capability without compromising motion imitation.”
Catch & Carry (2019, background): “Although Peng et al. 2019 reported that their multiplicative compositional policies (MCP) network structure performed better than an MLP in their settings, our own exploratory investigation of the reported MCP architecture (using 8 primitives) did not reproduce this benefit.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Recent work has developed mechanisms for skill chaining other than an explicit high-level policy, such as additively combining abstract skill embeddings (Devin et al., 2019) or multiplicatively combining policies (Peng et al., 2019).”
Toward Robust Long Range Policy Transfer (2021, background): “Some works propose a hierarchical policy (Frans et al. 2018; Bacon, Harb, and Precup 2017; Peng et al. 2019), which contains a combination function to control how to select or combine a set of primitives.”
Multi-Task Reinforcement Learning with Soft Modularization (2020, background): “Besides focusing on the gradients, another direction is to design compositional model with multiple modules (Singh, 1992; Devin et al., 2017; Andreas et al., 2017; Rusu et al., 2016; Qureshi et al., 2020; Peng et al., 2019; Haarnoja et al., 2018a; Sahni et al., 2017).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “…et al., 2013; Heess et al., 2016; Bacon et al., 2017; Florensa et al., 2017; Haarnoja et al., 2018a; Nachum et al., 2018; Chandak et al., 2019; Peng et al., 2019), and methods that learn temporally extended actions, also known as options , from demonstrations, and then recompose them to…”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, methodology): “…a multi-agent task from scratch (Lowe et al., 2017; Gupta et al., 2017; Sunehag et al., 2018; Foerster et al., 2018), modular approaches (Andreas et al., 2017; Frans et al., 2018; Peng et al., 2019) suggest to learn reusable primitive skills and solve more complex tasks by recombining these skills.”