Computing Factored Value Functions for Policies in Structured MDPs

By Daphne Koller, Ronald E. Parr. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, methodology): “Among value-based methods, value function factorization (Koller & Parr, 1999; Guestrin et al., 2002a; Sunehag et al., 2018; Rashid et al., 2018) methods have been proposed to efficiently handle a joint action-value function whose complexity grows exponentially with the number of agents.”
Optimal control as a graphical model inference problem (2009, positive): “Another promising approach is to exploit graphical structure that is present in the problem to make the computation more efficient (Boutilier et al. 1995; Koller and Parr 1999).”
Efficient Solution Algorithms for Factored MDPs (2003, methodology): “Our work builds on the ideas of Koller and Parr (1999, 2000), by using factored (linear) value functions, where each basis function is restricted to some small subset of the domain variables.”
Reinforcement Learning for RoboCup Soccer Keepaway (2005, background): “Koller & Parr, 1999; Guestrin, Koller, & Parr, 2001)).”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, background): “Value decomposition networks (VDN Sunehag et al., 2018) and QMIX (Rashid et al., 2018) are two representative examples of value function factorisation (Koller & Parr, 1999) that aim to efﬁciently learn a centralised but factored action-value function.”
Planning by Probabilistic Inference (2003, background): “Whereas much recent work has attempted to meet these challenges (see, e.g., [4{7]), planning under uncertainty remains an open problem and the subject of active research.”
Construction of approximation spaces for reinforcement learning (2013, background): “3 sufficiently close, factored MDP algorithms can be applied (Koller and Parr, 1999; Guestrin et al., 2001; Hauskrecht and Kveton, 2003; Guestrin et al., 2004).”
Probabilistic inference for solving discrete and continuous state Markov Decision Processes (2006, methodology): “For instance, in the ﬁeld of planning, they include work on Factored Markov Decision Processes (Boutilier et al., 1995; Koller & Parr, 1999; Guestrin et al., 2003; Kve-ton & Hauskrecht, 2005) or abstractions (Hauskrecht Appearing in Proceedings of the 23 rd International Conference on Machine…”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, methodology): “Value function factorisation [15] has been widely employed in value-based MARL algorithms.”
Efficient Reinforcement Learning in Factored MDPs (1999, methodology): “…the graphical structure (but not the parameters) of the DBN. Unlike the original E3 algorithm, our new algorithm exploits the DBN structure to achieve a running time that scales polynomially in the number ofparametersof the DBN, which may be exponentially smaller than the number of global states.”
Multiagent Planning with Factored MDPs (2001, methodology): “In [8], we proposed the use of factored linear value functions to approximate the value function in a factored MDP.”
A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes (1999, background): “approximation (Sutton & Barto, 1998) is a well-studied approach to learning value functions in large state spaces, and many authors have recently begun to study the properties of large MDPs that enjoy compact representations, such as MDPs in which the state transition probabilities factor into a sma”
Practical solution techniques for first-order MDPs (2009, methodology): “One general and popular approximate solution technique forMDPs is that of linearvalue function approximation (Schweitzer and Seidmann, 1985; Tsitsiklis and Van Roy, 1996; Koller and Parr, 1999, 2000; Schuurmans and Patrascu, 2001; Guestrin et al., 2002).”
Bayesian Time Series Models: Expectation maximisation methods for solving (PO)MDPs and optimal control problems (2011, methodology): “For instance, in the field of planning, they include work on Factored Markov Decision Processes (Boutilier et al., 1995; Koller and Parr, 1999; Guestrin et al., 2003; Kveton and Hauskrecht, 2005), abstractions (Hauskrecht et al.”
Q-Decomposition for Reinforcement Learning Agents (2003, background): “Other authors (Koller & Parr, 1999) have explored approximations that represent the true value function as a linear combination of basis functions, with each basis function deﬁned over a small collection of state variables.”