Learning values across many orders of magnitude

By Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, David Silver. 2016

205 citations recorded by the source corpus.

Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were all clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using the adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, background): “An initial foray into this problem is made in (van Hasselt et al. 2016), where the authors adaptively rescale reward targets with normalized stochastic gradient, but further research is needed.”
Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “Finally, van Hasselt et al. [19] successfully extended DQN to process unclipped rewards with an algorithm called PopArt, which adaptively rescales the targets for the value network to have zero mean and unit variance.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Pop-Art normalization (van Hasselt et al. 2016) allows reward clipping to be removed, while preserving a similar level of performance.”
Deep Q-learning From Demonstrations (2017, background): “…200 million steps with that of other deep reinforcement learning approaches: DQN, Double DQN, Prioritized DQN, Dueling DQN, PopArt, DQN+CTS, and DQN+PixelCNN (Mnih et al. 2015; van Hasselt, Guez, and Silver 2016; Schaul et al. 2016; Wang et al. 2016; van Hasselt et al. 2016; Ostrovski et al. 2017).”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “The double Q-learning update van Hasselt et al. (2016) uses the current network to calculate the argmax over next state values and the target network for the value of that action.”
Multi-task Deep Reinforcement Learning with PopArt (2018, methodology): “In this section we use PopArt normalisation (van Hasselt et al. 2016), which was introduced for value-based RL, to derive a scale invariant algorithm for actor-critic agents.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, methodology): “[3], and they were adopted as a primary source of stochasticity in most subsequent papers working in the Atari domain [3, 26, 27, 34, 38, 41, 42, 45, 67–73].”
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, background): “(10)This could be addressed in future work by modifying DQN as suggested by [27].”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Similarly to the the original V-MPO implementation, we use single-task PopArt normalisation of the value function (Hessel et al., 2019; van Hasselt et al., 2016).”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “(Hessel et al., 2018) uses PopArt normalization (van Hasselt et al., 2016) to perform loss weighting for multi-task deep reinforcement learning.”
Count-Based Exploration in Feature Space for Reinforcement Learning (2017, methodology): “Even algorithms that use sophisticated nonlinear methods for policy evaluation tend to use very old, inefﬁcient exploration techniques, such as the (cid:15) -greedy strategy [ van Hasselt et al. , 2016b; Mnih et al. , 2016; Nair et al. , 2015 ] .”
Deep Reinforcement Learning: An Overview (2017, background): “As a consequence, it is more likely to select overestimated values, and results in overoptimistic value estimates. van Hasselt et al. (2016a) proposed to evaluate the greedy policy according to the online network, but to use the target network to estimate its value.”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, methodology): “The Double Deep Q Network (DDQN) algorithm (van Hasselt et al., 2016b) uses the two separate networks that are already used in DQN for the separation such that the complexity of the algorithm is not increased.”
Machine Comprehension by Text-to-Text Neural Question Generation (2017, background): “The resulting “whitening” of the rewards is a simple version of PopArt (van Hasselt et al., 2016), and we found empirically that it stabilized learning.”