A Survey of Multi-Objective Sequential Decision-Making

By Diederik Marijn Roijers, Peter Vamplew, Shimon Whiteson, Richard Dazeley. 2013

837 citations recorded by the source corpus.

Sequential decision-making problems with multiple objectives arise naturally in practice and pose unique challenges for research in decision-theoretic planning and learning, which has largely focused on single-objective settings. This article surveys algorithms designed for sequential decision-making problems with multiple objectives. Though there is a growing body of literature on this subject, little of it makes explicit under what circumstances special methods are needed to solve multi-objective problems. Therefore, we identify three distinct scenarios in which converting such a problem to a single-objective one is impossible, infeasible, or undesirable. Furthermore, we propose a taxonomy that classifies multi-objective methods according to the applicable scenario, the nature of the scalarization function (which projects multi-objective values to scalar ones), and the type of policies considered. We show how these factors determine the nature of an optimal solution, which can be a single policy, a convex hull, or a Pareto front. Using this taxonomy, we survey the literature on multi-objective methods for planning and learning. Finally, we discuss key applications of such methods and outline opportunities for future work.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “For a more complete overview of MORL beyond the brief summary presented in this section, the interested reader is referred to recent surveys [58], [59].”
Challenges of Real-World Reinforcement Learning (2019, background): “For problems with multi-objective reward functions, there are approaches to learning the pareto-optimal reward function (Roijers et al., 2013), but none of these have been scaled to the deep reinforcement learning setting yet.”
Batch Policy Learning under Constraints (2019, background): “Generally, research in MORL has largely focused on approximating the Pareto frontier that trades-off competing objectives (Van Moffaert & Nowé, 2014; Roijers et al., 2013).”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “…these methods have been applied to multi-agent learning (Ghosh et al., 2013; Pirotta and Restelli, 2016; Parisi et al., 2014), kernel learning (Li et al., 2014), sequential decision making (Roijers et al., 2013), and Bayesian optimization (Shah and Ghahramani, 2016; Hernández-Lobato et al., 2016).”
A Distributional View on Multi-Objective Policy Optimization (2020, background): “An important quality of any MORL approach is the ability to find a variety of policies on the true Pareto front (Roijers et al., 2013).”
Multi-Objective Markov Decision Processes for Data-Driven Decision Support (2016, background): “(2010, 2012) and by many others (Roijers et al., 2013), we focus on the setting where the definition of an MDP is augmented by assuming a D-dimensional reward vector Rpst, atq is observed at each time step.”
Multi-objective reinforcement learning using sets of pareto dominating policies (2014, background): “In multi-objective optimization, the objective space consists of two or more dimensions (Roijers et al., 2013).”
A Lyapunov-based Approach to Safe Reinforcement Learning (2018, methodology): “This allows us to apply approaches such as the lexicographical ordering method from multi-objective stochastic optimal control methods [32] to solve the CMDP, for which the constraint value function has a higher lexicographical order than the objective value function.”
Fairness in Multi-Agent Sequential Decision-Making (2014, methodology): “When using centralized policies, our multi-agent MDPs can be also viewed as multi-objective MDPs [15].”
A practical guide to multi-objective reinforcement learning and planning (2021, methodology): “A linear utility function cannot accurately represent non-linear preferences [Roijers et al., 2013].”
Reward (Mis)design for Autonomous Driving (2021, background): “Multi-objective approaches [40] are an alternative to deﬁning a single utility or reward function for optimization.”
Urban Driving with Multi-Objective Deep Reinforcement Learning (2018, background): “An overview of multi-objective optmization can be found in [11], and a summary of multi-objective sequential decision making is given in [19].”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Ecologically-relevant extensions (Vamplew et al., 2021) include goal-conditioned RL (Liu et al., 2022) and multi-objective RL (Roijers et al., 2013), which allow the agent to consider utility functions that vary in a task-dependent fashion.”
Multiobjective Reinforcement Learning: A Comprehensive Overview (2015, background): “However, it covers the much broader topic of multiobjective sequential decision-making [18].”