Hybrid Reward Architecture for Reinforcement Learning

By Harm van Seijen, Mehdi Fatemi, Romain Laroche, Joshua Romoff, Tavian Barnes, Jeffrey Tsang. 2017

292 citations recorded by the source corpus.

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “…et al. (2016), is to add auxiliary prediction losses to the RL training criterion in order to encourage implicit learning of environment dynamics. van Seijen et al. (2017) obtain state of the art performance on MS PACMAN with a model free architecture, but they however rely on privileged…”
Safe Policy Improvement with an Estimated Baseline Policy (2019, background): “Additionally, they perform a benchmark on a continuous state space task, where the SPIBB and Soft-SPIBB algorithms signiﬁcantly outperform RaMDP and Double-DQN (Van Hasselt, Guez, and Silver 2016) both in reliability and average performance.”
Acquisition of chess knowledge in AlphaZero (2021, background): “Hierarchical reinforcement learning [57] and sub-task decomposition [58] can be seen as more interpretable, by introducing structure in the action space, where high-level goals are divided into sub-goals, and the corresponding sets of actions can therefore be more easily interpreted.”
Explainability in Deep Reinforcement Learning (2020, methodology): “Post-Hoc explainability refers to explainability methods that rely on an analysis done after the RL algorithm ﬁnishes its training and execution.”
Many-Goals Reinforcement Learning (2018, background): “3) Many-Goals Auxiliary Tasks: The third extension is to use many-goals updating as auxiliary control tasks to aid learning of the main task (cf. Mirowski et al. (2016); Van Seijen et al. (2017); Teh et al. (2017)).”
Mutual Alignment Transfer Learning (2017, background): “Recent work in reinforcement learning has led to significant successes such as outperforming humans on a multitude of computer games [1, 2] and surpassing the best human players in the games of Chess [3] and Go [4].”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “Indeed, their model combines the MADDPG algorithm with k-head Q-value model [142].”
Reinforcement Learning With Task Decomposition for Cooperative Multiagent Systems (2020, background): “Van Seijen et al. [23] studied the decomposed reward functions for multitask RL.”
Deep Reinforcement Learning: An Overview (2017, unknown): “See also Hadfield-Menell et al. (2016); Merel et al. (2017); Wang et al. (2017); van Seijen et al. (2017).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, background): “…exist that deal explicitly with the multi-objective nature of the learning problems, either by predicting a value function for each objective [Van Seijen et al. 2017], or by finding a policy that optimizes each sub-problem [Li et al. 2019], or that fits each Pareto-dominating mixture of…”
Recurrent Experience Replay in Distributed Reinforcement Learning (2018, methodology): “As an example, notice the performance on MS.PACMAN is even greater than that of the agent reported in (van Seijen et al., 2017), which was engineered specifically for this game.”
A benchmark environment motivated by industrial control problems (2017, background): “Pac-Man [15], and beating human experts in the game of Go [16].”
Learning to Compose Skills (2017, methodology): “Also related is work in multi-task RL, such as by van Seijen et al. [24], who use a Hybrid Reward Architecture agent to solve the game of Ms Pacman.”
Explainable Reinforcement Learning via Reward Decomposition (2019, background): “Recent work [16] considered reward decomposition for speeding up RL.”