A Distributional View on Multi-Objective Policy Optimization

By Abbas Abdolmaleki, Sandy H. Huang, Leonard Hasenclever, Michael Neunert, H. Francis Song, Martina Zambelli, Murilo Fernandes Martins, Nicolas Manfred Otto Heess, Raia Hadsell, Martin A. Riedmiller. 2020

94 citations recorded by the source corpus.

Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way. We propose to learn an action distribution for each objective, and we use supervised learning to fit a parametric policy to a combination of these distributions. We demonstrate the effectiveness of our approach on challenging high-dimensional real and simulated robotics tasks, and show that setting different preferences in our framework allows us to trace out the space of nondominated solutions.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Improving alignment of dialogue agents via targeted human judgements (2022, background): “A long-term approach should make the trade-off of helpfulness and harmlessness test-time configurable (Abdolmaleki et al., 2020) and train over an expanding universe of trade-offs and topics in an open ended fashion (Open Ended Learning Team et al., 2021) to find an optimal training data…”
Mastering Diverse Domains through World Models (2023, background): “Importantly, the network can output any continuous value in the interval because the weighted average can fall between the buckets: The network is trained on twohot encoded targets 8,28 , a generalization of onehot encoding to continuous values.”
A practical guide to multi-objective reinforcement learning and planning (2021, methodology): “[2] have addressed tasks with high-dimensional continuous action spaces using a multi-objective extension of the MPO actor-critic RL algorithm.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “(2020) uses an expectation maximization approach to learn multiple Q-functions per objective.”