NoRML: No-Reward Meta Learning

By Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Jie Tan, Chelsea Finn. 2019

29 citations recorded by the source corpus.

Efficiently adapting to new environments and changes in dynamics is critical for agents to successfully operate in the real world. Reinforcement learning (RL) based approaches typically rely on external reward feedback for adaptation. However, in many scenarios this reward signal might not be readily available for the target task, or the difference between the environments can be implicit and only observable from the dynamics. To this end, we introduce a method that allows for self-adaptation of learned policies: No-Reward Meta Learning (NoRML). NoRML extends Model Agnostic Meta Learning (MAML) for RL and uses observable dynamics of the environment instead of an explicit reward function in MAML's finetune step. Our method has a more expressive update step than MAML, while maintaining MAML's gradient based foundation. Additionally, in order to allow more targeted exploration, we implement an extension to MAML that effectively disconnects the meta-policy parameters from the fine-tuned policies' parameters. We first study our method on a number of synthetic control problems and then validate our method on common benchmark environments, showing that NoRML outperforms MAML when the dynamics change between tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, positive): “Meta-learning has been repeatedly used in the recent literature to learn functions for intrinsic motivation and improved exploration (Baranes and Oudeyer, 2009; Zheng et al., 2018; Stadie et al., 2018; Xu et al., 2018b; Houthooft et al., 2018; Yang et al., 2019; Zou et al., 2019; Zheng et al., 2019)”
Meta-Learning in Neural Networks: A Survey (2020, background): “Unsupervised meta-RL variants aim to perform meta-training without manually specified rewards [232], or adapt at meta-testing to a changed environment but without new rewards [233].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “To eliminate this difference, we can remove the inaccessible states during training (Tan et al. 2018), apply state estimation, add more sensors (e.g. Motion Capture) (Haarnoja et al. 2019) or learn to infer the missing information (e.g. reward) (Yang et al. 2019).”
Skill-based Meta-Reinforcement Learning (2022, background): “…al., 2019; Gupta et al., 2018; Vuorio et al., 2018; Nagabandi et al., 2019; Clavera et al., 2019; 2018; Rakelly et al., 2019; Vuorio et al., 2019; Yang et al., 2019; Zintgraf et al., 2019; Humplik et al., 2019; Zintgraf et al., 2020; Liu et al., 2021) hold the promise of allowing learning agents…”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Although researchers have developed principled approaches for mitigating the sim-toreal issue, including system identification [14, 31], domain randomization [45, 50, 47], and meta-learning [24, 53, 56], it remains an open problem.”