Orthographic Feature Transform for Monocular 3D Object Detection

By Thomas Roddick, Alex Kendall, Roberto Cipolla. British Machine Vision Conference, 2018

417 citations recorded by the source corpus.

3D object detection from monocular images has proven to be an enormously challenging task, with the performance of leading systems not yet achieving even 10\% of that of LiDAR-based counterparts. One explanation for this performance gap is that existing systems are entirely at the mercy of the perspective image-based representation, in which the appearance and scale of objects varies drastically with depth and meaningful distances are difficult to infer. In this work we argue that the ability to reason about the world in 3D is an essential element of the 3D object detection task. To this end, we introduce the orthographic feature transform, which enables us to escape the image domain by mapping image-based features into an orthographic 3D space. This allows us to reason holistically about the spatial configuration of the scene in a domain where scale is consistent and distances between objects are meaningful. We apply this transformation as part of an end-to-end deep learning architecture and achieve state-of-the-art performance on the KITTI 3D object benchmark.\footnote{We will release full source code and pretrained models upon acceptance of this manuscript for publication.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dfc700032ec2ec0b393c850fed61aa065ce7bb61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “Grid-based methods avoid estimating raw depth values by predicting a BEV grid [48, 55] representation, to be used as input for 3D detection architectures.”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Since there is only one front camera in KITTI, most of the methods have been developed to address monocular 3D detection [27, 38, 26, 18, 60, 62, 37, 47, 46].”
BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (2022, methodology): “Image features can also be used to generate 3D voxels [15] or orthographic feature maps [16].”
CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection (2020, methodology): “We compare with OFT [24], MonoDIS [26] and CenterNet [34] which are camera-based 3D object detection networks, as well as InfoFocus [27] which is a LIDAR-based method.”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, methodology): “The impact is smaller for the car class, but also in this case it is hard to resolve the difference between MonoDIS and OFT.”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, methodology): “Therefore, the monocular 3D detector cannot achieve satisfactory performance even some complex network structures [39] are applied.”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “For the nuScenes dataset, we compare against state-of-the-art models including CVT [3], FIERY [4], View Parsing Network (VPN) [27], Orthographic Feature Transform (OFT) [59], and Lift-SplatShoot [5].”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “[146] propose a Orthographic Feature Transform (OFT) algorithm to project the RGB image features onto the BEV plane.”
Disentangling Monocular 3D Object Detection (2019, methodology): “For the test set, we consider both the split in [3], which is shared with OFTNet [27] and ROI-10D [19], and a larger training split(1), since the setting used for MonoGRNet [24] is not clear.”
Probabilistic and Geometric Depth: Detecting Objects in Perspective (2021, methodology): “Transform to 3D representation Another category is to convert the RGB input to 3D representations like OFTNet [40] and Pseudo-Lidar [11].”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “We believe that our method could benefit from more research progress [26, 32, 33] in this direction.”
Delving into Localization Errors for Monocular 3D Object Detection (2021, methodology): “OFTNet [32] presents an orthographic feature transform to map image-based features into an orthographic 3D space.”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, background): “OFTNet [33] presents a orthographic feature transform to map image-based features into an orthographic 3D space.”
3D Object Detection for Autonomous Driving: A Survey (2021, background): “Consequently, instead of computing an intermediate point cloud, the other scheme wants to incorporate 3D geometry into image based networks and learn a latent depth distribution [4, 72, 74] explicitly or implicitly [70] in an end-to-end manner.”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, background): “IV-A, as well as their associated 3D LiDAR detections from ProbPIXOR [11] [115], or LiDAR points onto front-view for 3D detection [15].”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, background): “OFTNet [43] introduces the orthographic feature transform, which maps imagebased features into an orthographic 3D space.”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “BEVDet [17] and M(2)BEV [58] extends LSS [34] and OFT [41] to 3D object detection and CaDDN [38] adds explicit depth estimation supervision to the view transformer.”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “Orthographic Feature Transform (OFT) [33] creates a map-view intermediate representation from a monocular image by average pooling image features from the 2D projection that corresponds with the pillar in map-view.”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “OFT [39] and CaDDN [38] map the monocular image features into the bird’s eye view (BEV) and detect 3D objects in BEV space.”