Online Meta-Critic Learning for Off-Policy Actor-Critic Methods

By Wei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang, Timothy M. Hospedales. 2020

54 citations recorded by the source corpus.

Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. Normally, the critic's action-value function is updated using temporal-difference, and the critic in turn provides a loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared to the vanilla critic, the meta-critic network is explicitly trained to accelerate the learning process; and compared to existing meta-learning algorithms, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic framework is designed for off-policy based learners, which currently provide state-of-the-art reinforcement learning sample efficiency. We demonstrate that online meta-critic learning leads to improvements in avariety of continuous control environments when combined with contemporary Off-PAC methods DDPG, TD3 and the state-of-the-art SAC.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2019) and meta-critics (Zhou et al., 2020; Flennerhag et al., 2021) in a true continual RL setting closer to the spirit of self-modifying policies.”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “addressed the single-task setting, where the meta-knowledge such as loss [22], [180], reward [174], [181], hyperparameters [172], [173], or exploration strategy [182] are trained online together with the base policy while learning a single task.”
Discovering Reinforcement Learning Algorithms (2020, methodology): “This approach has been applied to meta-learn various forms of algorithmic components such as the discount factor [38], intrinsic rewards [41], auxiliary tasks [33], returns [36], auxiliary policy updates [42], off-policy corrections [39], and update target [37].”
Building a Subspace of Policies for Scalable Continual Learning (2022, unknown): “There is also a large body of work which leverages the shared structure of the tasks (Xu & Zhu, 2018; Pasunuru & Bansal, 2019; Lu et al., 2021; Mankowitz et al., 2018; Abel et al., 2017; Sodhani et al., 2021), meta-learning (Javed & White, 2019; Spigler, 2019; Beaulieu et al., 2020; Caccia et al., 2”