Policy invariance under reward transformations for multi-objective reinforcement learning
By Patrick Mannion, Sam Devlin, Karl Mason, Jim Duggan, Enda Howley. 2017
57 citations recorded by the source corpus.
Reinforcement Learning (RL) is a powerful and well-studied Machine Learning paradigm, where an agent learns to improve its performance in an environment by maximising a reward signal. In multi-objective Reinforcement Learning (MORL) the reward signal is a vector, where each component represents the performance on a different objective. Reward shaping is a well-established family of techniques that have been successfully used to improve the performance and learning speed of RL agents in single-objective problems. The basic premise of reward shaping is to add an additional shaping reward to the reward naturally received from the environment, to incorporate domain knowledge and guide an agent’s exploration. Potential-Based Reward Shaping (PBRS) is a specific form of reward shaping that offers additional guarantees. In this paper, we extend the theoretical guarantees of PBRS to MORL problems. Specifically, we provide theoretical proof that PBRS does not alter the true Pareto front in both single- and multi-agent MORL. We also contribute the first published empirical studies of the effect of PBRS in single- and multi-agent MORL problems.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Diederik Marijn Roijers
- Patrick Mannion
- Fredrik Heintz
- Marcello Restelli
- Peter Vamplew
- Richard Dazeley
- Luisa M. Zintgraf
- Enda Howley
In Computational Theory and Mathematics
- Patrick Mannion
- Diederik Marijn Roijers
- Marcello Restelli
- Peter Vamplew
- Richard Dazeley
- Luisa M. Zintgraf
- Enda Howley
- Ann Nowé
Citations in the corpus
- A Theoretical and Empirical Analysis of Reward Transformations in Multi-Objective Stochastic Games (2017, unknown): “1 shows that the best non-dominated solutions found by D and G+ PBRS(Mid) match closely with those of the true Pareto front.”
- A practical guide to multi-objective reinforcement learning and planning (2021, methodology): “Another frequently used model is the cooperative multi-objective stochastic game (MOSGs), where reinforcement learning or evolutionary algorithms were used to derive coverage sets (e.g., [Mannion et al., 2018, 2017, Yliniemi and Tumer, 2016]).”