Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks

By Thomas Roddick, Roberto Cipolla. Computer Vision and Pattern Recognition, 2020

308 citations recorded by the source corpus.

Autonomous vehicles commonly rely on highly detailed birds-eye-view maps of their environment, which capture both static elements of the scene such as road layout as well as dynamic elements such as other cars and pedestrians. Generating these map representations on the fly is a complex multi-stage process which incorporates many important vision-based elements, including ground plane estimation, road segmentation and 3D object detection. In this work we present a simple, unified approach for estimating these map representations directly from monocular images using a single end-to-end deep learning architecture. For the maps themselves we adopt a semantic Bayesian occupancy grid framework, allowing us to trivially accumulate information over multiple cameras and timesteps. We demonstrate the effectiveness of our approach by evaluating against several challenging baselines on the NuScenes and Argoverse datasets, and show that we are able to achieve a relative improvement of 9.1% and 22.3% respectively compared to the best-performing existing method.

Subfields: Computer Vision and Pattern Recognition, Ocean Engineering

View Paper: https://www.semanticscholar.org/paper/c7d59fe1b7b62886b0f7cd7239f9749c2e2ed2d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Ocean Engineering

Citations in the corpus

OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Considering the dimension gap between the 2D image input and the 3D prediction, recent studies for vision-based 3D perception first construct the BEV feature representations and then perform various downstream tasks on the BEV space [19, 28, 30, 37, 58, 38, 60, 40, 18, 1, 42].”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “[32], serves as the main comparison to prior work.”
FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras (2021, background): “Many previous works [30, 35, 39, 37, 42] have proposed a model to output the dynamic scene bird’s-eye view segmentation from multiview camera images of a single timeframe.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “Inspired by the design of LiDAR-based detectors, another type of cameraonly 3D perception models explicitly converts the camera features from perspective view to the bird’s-eye view using a view transformer [5], [38], [39], [6].”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, unknown): “OGM can be solved within a Bayesian framework, some recent works further combine semantic segmentation with OGM for downstream tasks [17, 42, 37].”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “We believe that our method could benefit from more research progress [26,32,33] in this direction.”
BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View (2021, methodology): “The methods with leading performance in benchmark [1] are always with the same perception pipeline [34, 30, 28, 47].”
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (2020, methodology): “We also outperform concurrent work that benchmarks on the same segmentation tasks [9] [28].”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “Images can be used to either directly estimate BEV semantic maps [39, 35, 60] or to estimate a BEV feature representation [44, 47, 41] as an intermediate step for the segmentation task.”
HDMapNet: An Online HD Map Construction and Evaluation Framework (2021, methodology): “Roddick and Cipolla [28] applied 1D convolution on the image fetures along the horizontal axis to predict bird’s-eye view.”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, background): “It is the up-todate popular benchmark for 3D object detection [37, 39, 38, 26] and BEV semantic segmentation [29, 27, 25, 41].”