TarMAC: Targeted Multi-Agent Communication

By Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael G. Rabbat, Joelle Pineau. 2018

534 citations recorded by the source corpus.

We propose a targeted communication architecture for multi-agent reinforcement learning, where agents learn both what messages to send and whom to address them to while performing cooperative tasks in partially-observable environments. This targeting behavior is learnt solely from downstream task-specific reward without any communication supervision. We additionally augment this with a multi-round communication approach where agents coordinate via multiple rounds of communication before taking actions in the environment. We evaluate our approach on a diverse set of cooperative multi-agent tasks, of varying difficulties, with varying number of agents, in a variety of environments ranging from 2D grid layouts of shapes and simulated traffic junctions to 3D indoor environments, and demonstrate the benefits of targeted and multi-round communication. Moreover, we show that the targeted communication strategies learned by agents are interpretable and intuitive. Finally, we show that our architecture can be easily extended to mixed and competitive environments, leading to improved performance and sample complexity over recent state-of-the-art approaches.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “Among those works, TarMAC falls in the explicit communication category [5], and ATOC, which is categorized as a implicit communication method, does not use graph convolution.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “…including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be better explained from a MARL…”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, methodology): “We use QMIX as representation for full factorization method; (ii) TarMAC (Das et al., 2019).”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, background): “Learning to communicate in a multi-agent setting has gained interest with the emergence of deep reinforcement learning [9, 10, 22, 4, 15, 7, 6].”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “…et al., 2018; Grover et al., 2018; Kipf et al., 2018) and explicit communication (Sukhbaatar et al., 2016; Hoshen, 2017; Jiang & Lu, 2018; Singh et al., 2019; Das et al., 2019; Singh et al., 2019; Kim et al., 2019) are studied to model the dependence between the decision-making processes of agents.”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “In the literature, communication policies can be either predefined or learned, allowing communication with all other agents or only a subset of agents.”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Visual, multi-agent, collaborative tasks have not been studied until very recently [23,42].”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, background): “Furthermore, some existing work tries to make use of communication to promote cooperation, such as (Sukhbaatar et al., 2016; Das et al., 2019; Singh et al., 2019).”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “Other works focused on methods that improves communication efficiency (Jiang & Lu, 2018; Das et al., 2019; Wang et al., 2021; Wan et al., 2022), cooperation in visually rich domains (Jain et al., 2020), or grounding communications in environments (Patel et al., 2021; Mandi et al., 2023;…”
Learning Distilled Collaboration Graph for Multi-Agent Perception (2021, background): “For example, CommNet [32] adopted the averaging operation to achieve continuous communication in multi-agent system; VAIN [11] considered an attention mechanism to determine which agents would share information; ATOC [12] exploited an attention unit to determine what information to share with other ”
Learning Transferable Cooperative Behavior in Multi-Agent Teams (2019, methodology): “In work done concurrently to ours, TarMAC (Das et al., 2018) uses dot-product attention mechanism for inter-agent communication, however, they do not impose any restrictions on communication, leading to a centralized execution paradigm.”
Succinct and Robust Multi-Agent Communication With Temporal Message Control (2020, methodology): “In this section, we evaluate TMC on Predator-Prey (PP) and Cooperative Navigation (CN), two popular MARL benchmarks which have been widely used by MARL community [19, 12, 5].”
Multi-agent deep reinforcement learning: a survey (2021, background): “The work by Das et al. (2019) introduced targeted multi-agent communication (TarMAC) that uses attention to decide with whom and what to communicate by actively addressing other agents for message passing.”
When2com: Multi-Agent Perception via Communication Graph Grouping (2020, methodology): “Recent works have proposed to construct communication groups based on pre-deﬁned rules [18, 17] or a uniﬁed communication network [33, 34, 15, 28, 33, 6].”
Learning Multi-Agent Communication through Structured Attentive Reasoning (2020, methodology): “3 Traffic Junction Task Similar to [14], [15], [16] we perform an evaluation on Traffic Junction, with 6, 10 and 20 agents respectively.”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “…agents (Jaques et al., 2019), optimizing both efﬁciency and fairness (Jiang & Lu, 2019), incorporating agent communication (Sukhbaatar et al., 2016; Das et al., 2019; Ding et al., 2020), considering relation between agents (Jiang et al., 2020) or their underlying network (Zhang et al., 2018; Qu et…”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, background): “Communication methods [19, 20, 21] share information between agents for better cooperation.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “…a computational framework to study many problems, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018).”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “[30] proposes an algorithm, called TarMAC, to learn to communicate in a multi-agent setting, where the agents learn to what to sent and also learn to communicate to which agent.”
Audio-Visual Embodied Navigation (2019, methodology): “Visual navigation can be tied with other tasks to attain intelligent behavior, such as question answering [23, 25, 39, 99], active visual recognition [50, 110, 109], instruction following [88, 6, 86, 17, 16] and collaborative tasks with visual inputs [49, 48, 24].”