CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

By Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, Jiaqi Ma. Conference on Robot Learning, 2022

398 citations recorded by the source corpus.

Bird's eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress on BEV map understanding, they are all based on single-agent camera-based systems. These solutions sometimes have difficulty handling occlusions or detecting distant objects in complex traffic scenes. Vehicle-to-Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems. In this paper, we propose CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. To efficiently fuse camera features from multi-view and multi-agent data in an underlying Transformer architecture, we design a fused axial attention module (FAX), which captures sparsely local and global spatial interactions across views and agents. The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation. Moreover, CoBEVT is shown to be generalizable to other tasks, including 1) BEV segmentation with single-agent multi-camera and 2) 3D object detection with multi-agent LiDAR systems, achieving state-of-the-art performance with real-time inference speed. The code is available at https://github.com/DerrickXuNu/CoBEVT.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/56e37466914419b2b93cf0dbe9343c49e968c066

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Ming-Hsuan Yang
2. Bolei Zhou
3. Peyman Milanfar
4. A. Bovik
5. Hao Xiang
6. Jiaqi Ma
7. Xin Xia
8. Runsheng Xu

In Computer Vision and Pattern Recognition

1. Ming-Hsuan Yang
2. Luc Van Gool
3. Dacheng Tao
4. Lei Zhang
5. Chen Change Loy
6. Xiaogang Wang
7. Trevor Darrell
8. Bernt Schiele

Citations in the corpus

• MaxViT: Multi-Axis Vision Transformer (2022, background): “More recent works have been focused on improving model and data efficiency, including sparse attention [1,21,64,86,96,99], improved locality [27,101], pyramidal designs [24,87, 97], improved training strategies [3,81,82,105], etc.”
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, methodology): “To ameliorate this issue, numerous methods have introduced locality into self-attention, such as Swin [25], CSwin [7], Twins [6], window [40,34], and sparse attention [37,35,43].”
• V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, methodology): “Similar to the cooperative detection track, CoBEVT [47] achieves the best performance in most of the evaluation metrics, including AMOTA (16.”