Improving Policy Gradient by Exploring Under-appreciated Rewards

By Ofir Nachum, Mohammad Norouzi, Dale Schuurmans. 2016

45 citations recorded by the source corpus.

This paper presents a novel form of policy gradient for model-free reinforcement learning (RL) with improved exploration properties. Current policy-based methods use entropy regularization to encourage undirected exploration of the reward landscape, which is ineffective in high dimensional spaces with sparse rewards. We propose a more directed exploration strategy that promotes exploration of under-appreciated reward regions. An action sequence is considered under-appreciated if its log-probability under the current policy under-estimates its resulting reward. The proposed exploration strategy is easy to implement, requiring small modifications to an implementation of the REINFORCE algorithm. We evaluate the approach on a set of algorithmic tasks that have long challenged RL methods. Our approach reduces hyper-parameter sensitivity and demonstrates significant improvements over baseline methods. Our algorithm successfully solves a benchmark multi-digit addition task and generalizes to long sequences. This is, to our knowledge, the first time that a pure RL method has solved addition using only reward feedback.

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Recent work has noted some connections between value and policy based RL (Norouzi et al., 2016; O’Donoghue et al., 2017; Nachum et al., 2017), when an entropy regularized objective is considered.”
Neural Architecture Search Using Deep Neural Networks and Monte Carlo Tree Search (2018, neutral): “However, directly maximizing the expected reward in Policy Gradient could lead to local optimal solution (Nachum, Norouzi, and Schuurmans 2016).”
Transfer Learning with Neural AutoML (2018, methodology): “We tried four variants of policy gradient to train the controller: REINFORCE [33], TRPO [34], UREX [35] and PPO [36].”
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood (2017, background): “, 2014, 2016), and methods which prioritize learning environment dynamics (Duff, 2002) or under-explored states (Kearns and Singh, 2002; Bellemare et al., 2016; Nachum et al., 2016).”
Neural Program Synthesis with Priority Queue Training (2018, methodology): “Nachum et al. (2017a) applies PG to program synthesis, but they differ from us in that they train RNNs that implicitly model a program by consuming inputs and emitting machine instructions as opposed to explicit programs.”
Stein Variational Policy Gradient (2017, background): “Recent work (Nachum et al., 2016) has explicitly attempted to explore ’under-appreciated’ rewards through importance sampling of trajectories over the current policy.”
AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search (2019, background): “However, directly maximizing the expected reward in vanilla Policy Gradient could lead to local optimal solution [29].”
Learning To Reach Goals Without Reinforcement Learning (2019, background): “…reward-weighted regression (Peters & Schaal, 2007), path-integral policy improvement (Theodorou et al., 2010), reward-augmented maximum likelihood (Norouzi et al., 2016; Nachum et al., 2016), and proportional cross-entropy method (Goschin et al., 2013) selectively weight policies or trajectories…”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “Direct policy search methods [Mannor et al., 2003, Peters and Schaal, 2007, Theodorou et al., 2010, Goschin et al., 2013, Norouzi et al., 2016, Nachum et al., 2016] selectively weight policies or trajectories by their performance during learning, as measured by the environment’s reward function or…”
Deep Reinforcement Learning: An Overview (2017, background): “There are usually two categories: chat-oriented and task-oriented systems; the former aims to converse with users in contextually reasonable way; the latter aims to assist users for speciﬁc goals (Su et al., 2016b).”
Learning to Generalize from Sparse and Underspecified Rewards (2019, background): “Inspired by (Norouzi et al., 2016; Nachum et al., 2017), we ﬁrst note that the IML objective per context x can be expressed in terms of a KL divergence between an optimal policy π ∗ and the parametric policy π , i.e., KL ( π ∗ π ) , whereas the RER objective per context x can be expressed in terms…”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “Additionally, we also describe the details of other algorithms [30, 49, 50, 41, 56, 36, 13, 52, 54] from the EM and KL control perspective.”