Learning to Explore with Meta-Policy Gradient

By Tianbing Xu, Qiang Liu, Liang Zhao, Jian Peng. 2018

54 citations recorded by the source corpus.

The performance of off-policy learning, including deep Q-learning and deep deterministic policy gradient (DDPG), critically depends on the choice of the exploration policy. Existing exploration methods are mostly based on adding noise to the on-going actor policy and can only explore \emph{local} regions close to what the actor policy dictates. In this work, we develop a simple meta-policy gradient algorithm that allows us to adaptively learn the exploration policy in DDPG. Our algorithm allows us to train flexible exploration behaviors that are independent of the actor policy, yielding a \emph{global exploration} that significantly speeds up the learning process. With an extensive study, we show that our method significantly improves the sample-efficiency of DDPG on a variety of reinforcement learning tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “To this end, potential directions aligned with this process of discovery through computation include: self-tuning approaches (Xu et al., 2018a; Zahavy et al., 2020), end-to-end skill discovery (Bacon et al., 2017), discovering RL algorithms (Kirsch, van Steenkiste, and Schmidhuber, 2019; Oh,…”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “…methods, gradient-based meta-RL methods learn from aggregated experience using policy gradients (Finn et al., 2017; Stadie et al., 2018; Rothfuss et al., 2018; Xu et al., 2018a), meta-learned loss functions (Sung et al., 2017; Houthooft et al., 2018), or hyperparameters (Xu et al., 2018b).”
Meta-Learning in Neural Networks: A Survey (2020, background): “addressed the single-task setting, where the meta-knowledge such as loss [22], [180], reward [174], [181], hyperparameters [172], [173], or exploration strategy [182] are trained online together with the base policy while learning a single task.”
Small Sample Learning in Big Data Era (2018, background): “, 2016), learning to reinforcement learn (Wang et al., 2016; Duan et al., 2016; Xu et al., 2018), learning to transfer (Ying et al.”
Efficient Exploration via State Marginal Matching (2019, background): “Some prior work has explicitly looked at the problem of learning to explore (Gupta et al., 2018; Xu et al., 2018).”
Meta Reinforcement Learning with Task Embedding and Shared Policy (2019, methodology): “, 2018], the replay buffer for training DDPG [Lillicrap et al., 2015; Xu et al., 2018b], non-stationary dynamics [Al-Shedivat et al.”
Multitask Soft Option Learning (2019, background): “et al., 1999; Konidaris & Barto, 2006), curriculum learning (Bengio et al., 2009), some meta-learning algorithms (Wang et al., 2016; Duan et al., 2016; Gupta et al., 2018; Houthooft et al., 2018; Xu et al., 2018), and transfer learning (Caruana, 1997; Taylor & Stone, 2011; Bengio, 2012; Pari”
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning (2018, methodology): “[45] use meta-learning to significantly speed up training by encouraging exploration of the state space outside of what the actor’s policy dictates.”
Dual Policy Distillation (2020, unknown): “Meta-learning methods also make use of a teacher model to improve the sample efﬁciency [Xu et al. , 2018b; Xu et al. , 2018a; Zha et al. , 2019b].”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “[40], both Eadvice 0:T and E 0:T are added to the task-level memories.”
What Can Learned Intrinsic Rewards Capture? (2019, methodology): “…our work is closely related to the prior work on meta-learning good exploration strategies (Wang et al., 2016; Duan et al., 2016; Stadie et al., 2018; Xu et al., 2018a) in that both perform temporal credit assignment across episode boundaries by maximising rewards accumulated beyond an episode.”
Learning to Generalize from Sparse and Underspecified Rewards (2019, methodology): “…interest in the ﬁeld of meta-reinforcement learning with previous work tackling problems such as automatically acquiring intrinsic motivation (Zheng et al., 2018), discovering exploration strategies (Gupta et al., 2018; Xu et al., 2018b), and adapting the nature of returns in RL (Xu et al., 2018c).”