Hierarchical visuomotor control of humanoids

By Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Manfred Otto Heess, Greg Wayne. 2018

102 citations recorded by the source corpus.

We aim to build complex humanoid agents that integrate perception, motor control, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies. The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. For a supplementary video link, see this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Critic Regularized Regression (2020, background): “[21] Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “Data description Note that for the purposes of data collection on the CMU humanoid tasks, we use expert policies trained according to Merel et al. (2019b), with only a single motor skill module from motion capture that is reused in each task.”
Neural probabilistic motor primitives for humanoid control (2018, methodology): “The approach we use for producing experts is detailed more fully in Merel et al. (2018).”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, background): “In the “gaps” task the 56-dimensional humanoid must run forward to match a target velocity of 4 m/s and jump over the gaps between platforms by learning to actuate joints with position-control (Merel et al., 2019).”
From Motor Control to Team Play in Simulated Humanoid Football (2021, methodology): “It extends the environment suite of (23, 31, 33, 42) and comprises teams of fully articulated humanoid”
dm_control: Software and Tasks for Continuous Control (2020, methodology): “The Locomotion framework has enabled a number of research eﬀorts including Merel et al. (2017), Merel et al. (2019b), Merel et al. (2019a) and more recently has been employed to support Multi-Agent domains in Liu et al. (2019), Sunehag et al. (2019) and Banarse et al. (2019).”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, background): “Others have grounded options in tasks [11] or complex action spaces [23].”
ANYmal parkour: Learning agile navigation for quadrupedal robots (2023, background): “While the locomotion policies are trained on single obstacles, the perception and navigation modules are trained on various arrangements of these obstacles.”
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies (2019, background): “A common approach for building hierarchical policies is to first train a collection of primitives through a set of pre-training tasks, which encourages each primitive to specialize in distinct skills [6, 12, 24, 25, 31].”
Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives (2019, background): “Auxiliary tasks such as motion imitation have been applied to learn motor primitives that are capable of performing a variety of sophisticated skills [20, 28, 23, 22].”
Deep neuroethology of a virtual rodent (2019, methodology): “…deep reinforcement learning algorithms which now allow artificial agents to perform complex and adaptive movement in physical environments with sensory information that is increasingly similar to that available to animals (Peng et al., 2016; 2017; Heess et al., 2017; Merel et al., 2019a;c).”
Deep Hierarchical Planning from Pixels (2022, unknown): “…via mutual-information maximization (Gregor et al., 2016; Florensa et al., 2017; Hausman et al., 2018; Eysenbach et al., 2018; Achiam et al., 2018; Merel et al., 2018; Laskin et al., 2022; Sharma et al., 2019; Xie et al., 2020; Hafner et al., 2020b; Strouse et al., 2021), or training the…”
Catch & Carry (2019, unknown): “Similarly to [Merel et al. 2019a], we provide the agent a normalized tracking reward, rt ∈ (0, 1], that reflects how well the body and object in the virtual environment match the reference:”
RL Unplugged: A Suite of Benchmarks for Ofﬂine Reinforcement Learning (2020, methodology): “These tasks are made up of the corridor locomotion tasks involving the CMU Humanoid [Tassa et al., 2020], for which prior efforts have either used motion capture data [Merel et al., 2019a,b] or training from scratch [Song et al., 2020].”
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future (2020, background): “Several such artificial systems have used convolutional architectures on the front end to process visual information about the world (Figure 3; Merel et al., 2018; Paine et al., 2018; Zhu et al., 2018).”
Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning (2019, background): “The behaviors and the selection rule can be learned separately [16], [14] or simultaneously [8], [19].”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, background): “…(Schmidhuber, 1990; Bacon et al., 2017; Nachum et al., 2018; Levy et al., 2019), meta-learned (Frans et al., 2018), pre-deﬁned (Kulkarni et al., 2016; Oh et al., 2017; Merel et al., 2019; Lee et al., 2019), or attained from additional supervision signals (Andreas et al., 2017; Ghosh et al., 2018).”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, methodology): “Similarly, techniques developed in the hierarchical reinforcement learning literature (e.g., Bacon et al., 2017; Dayan and Hinton, 1993; Florensa et al., 2017a; Haarnoja et al., 2018; Hausman et al., 2018; Merel et al., 2019; Nachum et al., 2019; Sutton et al., 1999) focus on structured behavior rep”