A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

By Runzhe Yang, Xingyuan Sun, Karthik Narasimhan. Neural Information Processing Systems, 2019

380 citations recorded by the source corpus.

We introduce a new algorithm for multi-objective reinforcement learning (MORL) with linear preferences, with the goal of enabling few-shot adaptation to new tasks. In MORL, the aim is to learn policies over multiple competing objectives whose relative importance (preferences) is unknown to the agent. While this alleviates dependence on scalar reward design, the expected return of a policy can change significantly with varying preferences, making it challenging to learn a single model to produce optimal policies under different preference conditions. We propose a generalized version of the Bellman equation to learn a single parametric representation for optimal policies over the space of all possible preferences. After an initial learning phase, our agent can execute the optimal policy under any given preference, or automatically infer an underlying preference with very few samples. Experiments across four different domains demonstrate the effectiveness of our approach.

View Paper: https://www.semanticscholar.org/paper/671441f065e0efe26bdcd18296f4743ae3d2d72a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Recent advances in reinforcement learning in finance (2021, background): “The study of multiobjective RL is still at a preliminary stage and relevant references include Zhou et al. (2020) and Yang et al. (2019).”
A Distributional View on Multi-Objective Policy Optimization (2020, methodology): “…approaches learn a set of policies simultaneously, by using a multi-objective variant of the Q-learning update rule (Barrett & Narayanan, 2008; Moffaert & Nowé, 2014; Reymond & Nowé, 2019; Yang et al., 2019) or by modifying gradientbased policy search (Parisi et al., 2014; Pirotta et al., 2015).”
Effective Diversity in Population-Based Reinforcement Learning (2020, methodology): “To test this we create environments where an agent can be rewarded for multiple different behaviors, as is typically done in multi-objective RL (Yang et al., 2019).”
A practical guide to multi-objective reinforcement learning and planning (2021, methodology): “The Coverage Ratio metric is used in [Yang et al., 2019] as an evaluation metric for comparing different multi-objective algorithms.”