Optimal Payoff Functions for Members of Collectives

By David Hilton Wolpert, Kagan Tumer. 2001

308 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Sociology and Political Science, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Sociology and Political Science

In Game & Decision Theory

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, background): “The idea is inspired by difference rewards (Wolpert and Tumer 2002; Tumer and Agogino 2007), in which each agent learns from a shaped reward that compares the global reward to the reward received when that agent’s action is replaced with a default action.”
Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “As pointed out by [23], one has to conduct counterfactual experiments”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, neutral): “Later, Foerster et al. (2018) proposed Counterfactual Multi-Agent Policy Gradients (COMA) motivated from the difference reward mechanism (Wolpert & Tumer, 2001) to address the challenges of multi-agent credit assignment.”
Multiagent Learning: Basics, Challenges, and Prospects (2012, background): “…learn to behave optimally in the presence of other (learning) agents, facing the complexities of incomplete information, large state spaces, credit assignment, cooperative or competitive settings, and reward shaping (Wolpert and Tumer 2001; Tumer, Agogino, and Wolpert 2002; Agogino and Tumer 2008).”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, background): “The probability of each reward is 50% . cooperative multi-agent systems in which agents acting to optimize their own agent rewards will also optimize the provided global reward (Wolpert & Tumer, 2001; Agogino & Turner, 2005).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “, the moving target problem (non-stationarity) [55,141,289], curse of dimensionality [55,289], multiagent credit assignment [2,355], global exploration [213], and relative overgeneralization [105,247,347].”
Distributed agent-based air traffic flow management (2007, background): “The second reward was a personalized agent reward based on collectives [2, 16 , 17].”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “We use the aristocrat utility (Wolpert & Tumer, 2002) to perform credit assignment: It is worth noting that U i is independent of other agents’ actions.”
Combining reward shaping and hierarchies for scaling to large multiagent systems (2016, unknown): “Difference rewards are a particular type of shaped rewards that were developed to address the structural credit assignment problem within multiagent systems (Wolpert & Tumer, 2001; Tumer, 2005; Combining reward shaping and hierarchies Agogino et al ., 2012; HolmesParker et al ., 2012).”
A multiagent approach to managing air traffic flow (2010, background): “In this work we focus on “difference rewards” which aim to provide a reward that is both sensitive to that agent’s actions and aligned with the overall system reward [2,49,53].”
A Survey of Collectives (2004, methodology): “In addition we have explored COIN-based techniques on variants of congestion games [238, 242, 243], in particular of a more challenging variant of Arthur’s El Farol bar attendance problem [5] (also known as the “minority game” [48]).”
Analyzing and visualizing multiagent rewards in dynamic and stochastic domains (2008, methodology): “Creating high-performance reward structures that promote coordination is a critical step in designing successful multiagent learning algorithms [8,15,18,21,22,26,31].”
Evolutionary Computation for Reinforcement Learning (2012, background): “The main idea is to use difference functions (Wolpert and Tumer, 2002) that compare the total reward the team obtains when the agent is present to when it is absent or replaced by a fixed baseline agent.”
Learning sequences of actions in collectives of autonomous agents (2002, methodology): “, "alignedness" and "Ieaxnability") one can use the "COllective INtelligence" (COIN) framework [19, 28].”
An Overview of Cooperative and Competitive Multiagent Learning (2005, background): “Similarly, Wolpert and Tumer [197] argue that global reward does not scale well to increasingly difficult problems because the learners do not have sufficient feedback tailored to their own specific actions.”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, methodology): “Such complex tasks require MARL to learn decentralized policies for agents to jointly optimize a global cumulative reward signal, and post a number of challenges, including multi-agent credit assignment [4–6], non-stationarity [7,8], and scalability [3,9].”