Cross-view Transformers for real-time Map-view Semantic Segmentation

By Brady Zhou, Philipp Krahenbuhl. Computer Vision and Pattern Recognition, 2022

388 citations recorded by the source corpus.

We present cross-view transformers, an efficient attention-based model for map-view semantic segmentation from multiple cameras. Our architecture implicitly learns a mapping from individual camera views into a canonical map-view representation using a camera-aware cross-view attention mechanism. Each camera uses positional embeddings that depend on its intrinsic and extrinsic calibration. These embeddings allow a transformer to learn the mapping across different views without ever explicitly modeling it geometrically. The architecture consists of a convolutional image encoder for each view and cross-view transformer layers to infer a map-view semantic segmentation. Our model is simple, easily parallelizable, and runs in realtime. The presented architecture performs at state-of-the-art on the nuScenes dataset, with 4x faster inference speeds. Code is available at https://github.com/bradyz/cross_view_transformers.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c5e6f0c52c1f91086879f46120efa79e96158eba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “Previous works [19, 21] use rasterized semantic maps to guide the planning, which misses critical instance-level structure information of the map.”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “This idea has also been adopted by several concurrent papers on BEV semantic segmentation from image inputs [81], [82], [83], [84], [85].”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “We take a BEV processing architecture similar to CVT [3], wherein a learnable BEV embedding is initialized as the query to interact with encoded multi-view camera features, as shown in Fig.”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, background): “The recent development of deep learning brings significant breakthroughs in various perception tasks such as 3D object detection [22, 35, 42], object tracking [43, 56], and semantic segmentation [47, 57].”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Following the previous method [69], we adopted CVT [72] for foreground and background segmentation.”
MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction (2022, methodology): “MapTR is compatible with other transformation methods and maintains stable performance, e.g. , CVT (Zhou & Kr¨ahenb¨uhl, 2022), LSS (Philion & Fidler, 2020; Liu et al., 2022c; Li et al., 2022b; Huang et al., 2021), Deformable Attention (Li et al., 2022c; Zhu et al., 2021) and IPM (Mallot et al.,…”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, methodology): “Different from the 2D-to-3D lifting in LSS or the 3D-to2D projection in OFT, CVT [43] utilized camera-aware positional encoding and dense cross attention to bridge perspective-view and BEV-view features.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “Then the updated seg queries are ﬁnally fed into the segmentation head, similar to the decoder in CVT [43], to predict the ﬁnal segmentation results.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “For each frame, we only perform the evaluation in the [-50m, 50m]×[-50m, 50m] region around the ego car following [6], [8], [41], [43].”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, background): “Recently, several cross-view learning approaches (Philion & Fidler, 2020; Pan et al., 2020; Li et al., 2021; Zhou & Kr¨ahenb¨uhl, 2022; Wang et al., 2022; Chen et al., 2022) leverage the geometric structure of scenes to mitigate the mismatch between sensor inputs and BEV representations.”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Considering the dimension gap between the 2D image input and the 3D prediction, recent studies for vision-based 3D perception first construct the BEV feature representations and then perform various downstream tasks on the BEV space [19, 28, 30, 37, 58, 38, 60, 40, 18, 1, 42].”