All learning is Local: Multi-agent Learning in Global Reward Games

By Yu-Han Chang, T. Ho, Leslie Pack Kaelbling. 2003

189 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, background): “Another crucial challenge is multi-agent credit assignment (Chang et al., 2003): in cooperative settings, joint actions typically generate only global rewards, making it difﬁcult for each agent to deduce its own contribution to the team’s success.”
Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning (2019, background): “Optimising a global objective also becomes challenging at scale, as it becomes difficult to assign credit to each agent [4, 35].”
An Overview of Cooperative and Competitive Multiagent Learning (2005, unknown): “Chang et al [33] take a different approach to perform credit assignment: each agent employs a Kalman filter to compute its true contribution to the global reward.”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “Associating rewards to agents is known as the credit assignment problem (Chang et al. 2004; Weiß 1995; Wolpert and Tumer 1999).”
Q-value Path Decomposition for Deep Multiagent Reinforcement Learning (2020, background): “Therefore, the key to promoting the coordination of agents is to correctly allocate the reward signal for each agent, which is also known as the multiagent credit assignment problem (Chang et al., 2004).”
On Multi-Agent Learning in Team Sports Games (2019, background): “In such a framework, iteratively optimizing for a policy could suffer from non-convergence due to the breakdown of the stationarity of the decision process and partial observability of the state space (Littman, 1994; Chang et al., 2004).”
Fully Decentralized Policies for Multi-Agent Systems: An Information Theoretic Approach (2017, background): “An additional difficulty in the multi-agent POMDP setting occurs when agents have to learn online from a common global reward signal, which effectively masks which agents bear responsibility for changes in reward [Chang et al., 2003].”
CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning (2018, background): “a global reward) [5, 26, 6, 22], and while one could treat the cooperative multi-goal scenario as a problem with a single joint goal, this coarse approach makes it extremely difficult to isolate the impact of any agent’s action on any”
Efficient Distributed Reinforcement Learning through Agreement (2008, background): “, [12, 3]) that the reward is a global scalar available unaltered to all agents1.”
Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks (2018, background): “…applied to both cooperative and competitive scenarios, (ii) it also helps resolve the credit assignment issue faced by many multi-agent (Sukhbaatar et al., 2016; Foerster et al., 2016a) algorithms while improving performance with scalability and is coherent with the ﬁndings in Chang et al. (2003).”
Decentralized Learning in Markov Games (2008, background): “In [ 21 ] the agents are not assumed to have a full view of the world.”
Cooperative Multi-Agent Learning: The State of the Art (2005, background): “[49] take a different approach to credit assignment: each agent assumes the observed reward signal is a sum of the agent’s direct contribution and some random Markov process that estimates the contributions of teammates.”
Human Reinforcement Learning Subdivides Structured Action Spaces by Learning Effector-Specific Values (2009, methodology): “If values are learned using prediction errors, as commonly supposed, then decomposed values are most efficiently learned using separate prediction errors (Chang et al., 2003).”
Hierarchical Multiagent Reinforcement Learning for Maritime Traffic Management (2020, background): “The policy trained with global empirical returnGt is known to be sample inefficient and results in poor solution quality particularly in multiagent settings because the global reward signal does not address the multiagent credit assignment problem [3, 7].”
Multiagent Learning and Coordination with Clustered Deep Q-Network (2019, background): “Decentralized policies face with another challenge: the multiagent credit assignment [2], where joint actions usually generate only global rewards, making it difficult for each agent to deduce its own contribution.”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, background): “…actions and observations with Q ⇡ ( o , a ) and V ⇡ ( o ) , deﬁned as The jointness of the objective causes difﬁculties associated with the credit assignment problem— having received a shared reward, individual agents are unable to deduce their own contribution to the team’s success or failure [4].”