V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer

By Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, Jiaqi Ma. European Conference on Computer Vision, 2022

684 citations recorded by the source corpus.

In this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel vision Transformer. Specifically, we build a holistic attention model, namely V2X-ViT, to effectively fuse information across on-road agents (i.e., vehicles and infrastructure). V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention, which captures inter-agent interaction and per-agent spatial relationships. These key modules are designed in a unified Transformer architecture to handle common V2X challenges, including asynchronous information sharing, pose errors, and heterogeneity of V2X components. To validate our approach, we create a large-scale V2X perception dataset using CARLA and OpenCDA. Extensive experimental results demonstrate that V2X-ViT sets new state-of-the-art performance for 3D object detection and achieves robust performance even under harsh, noisy environments. The code is available at https://github.com/DerrickXuNu/v2x-vit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7476e1c4fd6fa8e6d23bd7e1b45f9e799d04b284

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MAXIM: Multi-Axis MLP for Image Processing (2022, background): “Despite successful applications to many high-level tasks [4, 24, 57, 85, 89, 102, 104], the efficacy of these global models on lowlevel enhancement and restoration problems has not been studied extensively.”
Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps (2022, methodology): “We follow the localization noise setting in V2VNet and V2X-ViT (Gaussian noise with a mean of 0m and a standard deviation of 0m-0.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, neutral): “agent scenarios [257, 258, 259, 260, 261], these systems could provide a solution to realize advanced decision intelligence among autonomous vehicles.”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “Several prior works have demonstrated the efficacy of cooperative perception utilizing LiDAR sensors [12, 13, 14, 15].”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, methodology): “Most of the existing V2V datasets, including OPV2V [50], V2X-Sim [20], and V2XSet [49], rely on open-source simulators like CARLA [11] to generate synthetic road scenes and traffic dynamics with simulated connected vehicles.”