Who2com: Collaborative Perception via Learnable Handshake Communication

By Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma, Nathan Glaser, Chia-Wen Kuo, Zsolt Kira. 2020

261 citations recorded by the source corpus.

In this paper, we propose the problem of collaborative perception, where robots can combine their local observations with those of neighboring agents in a learnable way to improve accuracy on a perception task. Unlike existing work in robotics and multi-agent reinforcement learning, we formulate the problem as one where learned information must be shared across a set of agents in a bandwidth-sensitive manner to optimize for scene understanding tasks such as semantic segmentation. Inspired by networking communication protocols, we propose a multi-stage handshake communication mechanism where the neural network can learn to compress relevant information needed for each stage. Specifically, a target agent with degraded sensor data sends a compressed request, the other agents respond with matching scores, and the target agent determines who to connect with (i.e., receive information from). We additionally develop the AirSim-CP dataset and metrics based on the AirSim simulator where a group of aerial robots perceive diverse landscapes, such as roads, grasslands, buildings, etc. We show that for the semantic segmentation task, our handshake communication method significantly improves accuracy by approximately 20% over decentralized baselines, and is comparable to centralized ones using a quarter of the bandwidth.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

When2com: Multi-Agent Perception via Communication Graph Grouping (2020, methodology): “• Who2com [23] (DistCom) excludes self-attention mechanism such that it always communicates with one of the supporting agents.”
Learning Distilled Collaboration Graph for Multi-Agent Perception (2021, background): “To deal with the performance-bandwidth trade-off, intermediate collaboration [19, 34, 20] has been proposed to aggregate intermediate features across agents; see Fig.”
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, background): “For example, Who2Com [10] studies which agent to request/transmit data, largely reducing computation.”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Collaborative perception for semantic segmentation and recognition classification have also been investigated recently [55,56].”
V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving (2022, methodology): “The intermediate models, including DiscoNet [13], V2VNet [12], When2com [18], and Who2com [19], are based on the communication of the intermediate features of the neural network.”