Cross-View Semantic Segmentation for Sensing Surroundings

By Bowen Pan, Jiankai Sun, Ho-yin. Leung, Alex Andonian, Bolei Zhou. IEEE Robotics and Automation Letters, 2019

329 citations recorded by the source corpus.

Sensing surroundings plays a crucial role in human spatial perception, as it extracts the spatial configuration of objects as well as the free space from the observations. To facilitate the robot perception with such a surrounding sensing capability, we introduce a novel visual task called Cross-view Semantic Segmentation as well as a framework named View Parsing Network (VPN) to address it. In the cross-view semantic segmentation task, the agent is trained to parse the first-view observations into a top-down-view semantic map indicating the spatial location of all the objects at pixel-level. The main issue of this task is that we lack the real-world annotations of top-down-view data. To mitigate this, we train the VPN in 3D graphics environment and utilize the domain adaptation technique to transfer it to handle real-world data. We evaluate our VPN on both synthetic and real-world agents. The experimental results show that our model can effectively make use of the information from different views and multi-modalities to understanding spatial information. Our further experiment on a LoCoBot robot shows that our model enables the surrounding sensing capability from 2D image input. Code and demo videos can be found at https://view-parsing-network.github.io.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5333c412657391739eb4dd9e306ceed6def4d303

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “Although they have demonstrated remarkable performance on large-scale detection benchmarks, these point-level fusion methods barely work on semantic-oriented tasks, such as BEV map segmentation [5], [6], [7], [8].”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “BEV segmentation methods [42, 49] attempt to predict BEV semantic maps of 3D scenes from images.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, background): “Recently, several cross-view learning approaches (Philion & Fidler, 2020; Pan et al., 2020; Li et al., 2021; Zhou & Kr¨ahenb¨uhl, 2022; Wang et al., 2022; Chen et al., 2022) leverage the geometric structure of scenes to mitigate the mismatch between sensor inputs and BEV representations.”
Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks (2020, methodology): “Unfortunately, given the lack of available ground truth data, many of the above methods are forced to rely on weak supervision from stereo [17], weakly-aligned map labels [24] or synth-to-real domain transfer [24, 21].”
PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Recent works have shown that deep networks can learn to plan directly from pixel observations in domains such as table-top manipulation [12], [13], [14], navigation [15], [16], [17], and locomotion in joint space [18].”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, background): “It is the up-todate popular benchmark for 3D object detection [37, 39, 38, 26] and BEV semantic segmentation [29, 27, 25, 41].”
BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision (2022, methodology): “Early works including OFT [30], Pseduo LiDAR [36], and VPN [26] shed light on how to transform perspective features into BEV features but either for a single camera or on less well-known tasks.”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “Both VED and VPN show carefully-designed networks trained with sufﬁcient training data can jointly learn the map-view transformation and perform prediction.”
BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View (2021, methodology): “The methods with leading performance in benchmark [1] are always with the same perception pipeline [34, 30, 28, 47].”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “To eliminate the effect of task heads and compare other BEV generating methods fairly, we use VPN [30] and Lift-Splat [32] to replace our BEVFormer and keep task heads and other settings the same.”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “For the nuScenes dataset, we compare against state-of-the-art models including CVT [3], FIERY [4], View Parsing Network (VPN) [27], Orthographic Feature Transform (OFT) [59], and Lift-SplatShoot [5].”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, background): “Some learning based methods [12, 37] implicitly project image input into BEV, but the quality can not be guaranteed since usually we do not have ground truth in BEV to supervise the projection process.”
HDMapNet: An Online HD Map Construction and Evaluation Framework (2021, methodology): “VPN [27] proposes a simple view transformation module: a view relation module to model the relations between any two pixels and a view fusion module to fuses the features of pixels.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, background): “The distance is calculated at several fixed positions along the Y-axis: [5, 10, 15, 20, 30, 40, 50, 60, 80, 100] for 3D anchor lanes.”