Q-Decomposition for Reinforcement Learning Agents

By Stuart J. Russell, Andrew Zimdars. 2003

202 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “Russell and Zimdars (2003) sums the Q -functions of independent learning agents with individual rewards, before making the global action selection greedily to optimize for total reward.”
Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward (2018, background): “Schneider et al. [21] and Russell and Zimdars [20] optimize the sum of individual rewards by learning value functions from those individual rewards.”
A Survey of Multi-Objective Sequential Decision-Making (2013, background): “One key weakness of such approaches was pointed out by Russell and Zimdars (2003): they do not allow for the selection of actions that, while not optimal for any single objective, oﬀer a good compromise between multiple objectives.”
Sparse cooperative Q-learning (2004, background): “From this perspective, our local Q-learning updates seem closer in spirit to the local Sarsa updates of ( Russell & Zimdars, 2003 ).”
Multi-agent deep reinforcement learning: a survey (2021, background): “∏ Next to the policy, the value function can be factored so that the joint value is decomposed into a sum of local value functions, e.g. the joint action-value function can be expressed by Russell and Zimdars (2003).”
Transfer in variable-reward hierarchical reinforcement learning (2008, background): “Russell and Zimdars (2003) consider the additive decomposition of rewards to solve the MDP.”
Human Reinforcement Learning Subdivides Structured Action Spaces by Learning Effector-Specific Values (2009, background): “Work in computational RL (Chang et al., 2003; Russell and Zimdars, 2003) uses decomposition to simplify learning tasks such as controlling fishing by a fleet of boats.”
Object focused q-learning for autonomous agents (2013, unknown): “This makes our algorithm similar to modular reinforcement learning, even though we have different goals than modular RL. Russell & Zimdars [13] take into account the whole state space for the policy of each module , so they can obtain global optimality, at the expense of not addressing the dimension”
Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning (2018, background): “The overall idea is to train separate agents using different reward functions and then have an arbiter combine the agents’ opinions about which action to perform next (Karlsson, 1994; Russell & Zimdars, 2003; van Seijen et al., 2016).”
Multiagent-Based Reinforcement Learning for Optimal Reactive Power Dispatch (2012, background): “[11] proposed a Q-decomposition [12] based approach with a central arbitrator to coordinate the reward of separated agents and reduce the sets of states and actions to consider.”
Credit Assignment in Multiple Goal Embodied Visuomotor Behavior (2010, background): “…that, to a first approximation, such modules are activated in subsets whose members either do not interfere with each other (Guestrin et al., 2003; Russell and Zimdars, 2003; Sprague et al., 2007), or, if they do, then the interference can be handled in a way that approximates the result one…”
Optimized Computation Offloading Performance in Virtual Edge Computing Systems Via Deep Reinforcement Learning (2018, unknown): “(5)(χ,(c,e)), then K= {1,2,3,4}forms a classiﬁcation pattern. In our considered stochastic computation ofﬂoading scenario, it’s easy to see that K ≤5. 18 Mathematically, Q(χ,(c,e)) is decomposed into [29] Q(χ,(c,e)) = X k∈K Qk(χ,(c,e)), (33) where the MU deploys a “virtual” agent k ∈Kto learn the op”
A Distributional View on Multi-Objective Policy Optimization (2020, methodology): “2), inspired by Russell & Zimdars (2003).”
STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning (2025, unknown): “The construction of embodied agents can be categorized into three approaches, including rule-based and traditional planning methods [13], RL methods [14], and LLM-based methods [15].”