3D Packing for Self-Supervised Monocular Depth Estimation

By Vitor Campanholo Guizilini, Rares Ambrus, Sudeep Pillai, Adrien Gaidon. Computer Vision and Pattern Recognition, 2019

818 citations recorded by the source corpus.

Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception. In this work, we propose a novel self-supervised monocular depth estimation method combining geometry with a new deep network, PackNet, learned only from unlabeled monocular videos. Our architecture leverages novel symmetrical packing and unpacking blocks to jointly learn to compress and decompress detail-preserving representations using 3D convolutions. Although self-supervised, our method outperforms other self, semi, and fully supervised methods on the KITTI benchmark. The 3D inductive bias in PackNet enables it to scale with input resolution and number of parameters without overfitting, generalizing better on out-of-domain data such as the NuScenes dataset. Furthermore, it does not require large-scale supervised pretraining on ImageNet and can run in real-time. Finally, we release DDAD (Dense Depth for Automated Driving), a new urban driving dataset with more challenging and accurate depth evaluation, thanks to longer-range and denser ground-truth depth generated from high-density LiDARs mounted on a fleet of self-driving cars operating world-wide.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/43fcb6cf679cd2ae412d7cb53e85312600cc20dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Input and ground truth ManyDepth (MR) Monodepth2 [24] DFTP [66] Packnet-SFM [28] Input and ground truth ManyDepth (MR) Monodepth2 [24] DFTP [66] Packnet-SFM [28] Figure 9.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, background): “For example, when tested on the well-known autonomous driving dataset DDAD [20], we improve the Ab-sRel ( ↓ ) from 0.251 → 0.230 and improve the δ 1 ( ↑ ) from 0.766 → 0.789.”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, neutral): “12, including comic-style (Row 2) and CG(computer graphics)-generated objects (Row5)”
MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision (2024, neutral): “We evaluate the accuracy of point map estimation on eight diverse datasets: NYUv2 [35], KITTI [47], ETH3D [45], iBims-1 [29], Sintel [8], Google Scanned Objects (GSO) [15], DDAD [22], and DIODE [48].”
DUSt3R: Geometric 3D Vision Made Easy (2023, neutral): “We benchmark DUSt3R on two outdoor (DDAD [41], KITTI [35]) and three indoor (NYUv2 [115], BONN [80], TUM [119]) datasets.”
DSEC: A Stereo Event Camera Dataset for Driving Scenarios (2021, methodology): “Oxford RobotCar [27] and KAIST Urban [28] target localization and mapping, while DDAD [29] and DrivingStereo [6] are specialized for monocular or stereo depth estimation respectively.”
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth (2023, methodology): “For the outdoor datasets, we use DDAD [12], DIML Outdoor [17], DIODE Outdoor [40] and Virtual KITTI 2 [7].”
UniDepth: Universal Monocular Metric Depth Estimation (2024, methodology): “More precisely, each method is tested on validation splits from SUN-RGBD [38] without NYU split, Diode Indoor [40] , IBims-1 [22], VOID [44] HAM-MER [21], ETH-3D [37], nuScenes [4], and DDAD [19] with split proposed in [34] and evaluated with official masks.”
HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation (2020, methodology): “In addition, (Guizilini et al. 2020a) utilizes packing and unpacking block to preserve the spatial information in image and low level feature.”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, neutral): “We trained our metric depth model on 14 datasets, including Taskonomy [120], DIML (Outdoor) [15], DDAD [30], Argoverse [101], Lyft [], PandaSet [105], Waymo [84], ScanNet++ [117], ARK-itScenes [5], Map-free [3], DSEC [24], Driving Stereo [109] and Cityscapes [16] datasets.”
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries (2021, methodology): “As an alternative to these 2D-based methods, some methods incorporate more 3D computations into our object detection pipeline by applying a 3D reconstruction method like [4, 5, 6] to create a pseudo-LiDAR or range input of the scene from camera images.”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “lly-masked photometric loss adapting (3) Jphm t= ˝ tmin t02T0 2 (1 SSIM(x;x 0!)) + (1 )jx x j ˛ ; (9) which only considers non-DC pixels. We also consider the mask from the automasking technique [20,23,24], which is omitted in (3) and (9) for simplicity. Self-Supervised Depth Estimation With Semant”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, methodology): “Different kinds of backbone, such as VGGNet, ResNet, HRNet and PackNet, made their way into the self-supervised monocular depth estimation task [71, 17, 69, 18].”
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image (2023, methodology): “Datasets have [11] Outdoor LiDar ∼ 80K 36+ Lyft [12] Outdoor LiDar ∼ 50K 6+ Driving Stereo (DS) [23] Outdoor Stereo † ∼ 181K 1 DIML [5] Outdoor Stereo † ∼ 122K 10 Arogoverse2 [21] Outdoor LiDar ∼ 3515K 6+ Cityscapes [6] Outdoor Stereo † ∼ 170K 1 DSEC [9] Outdoor LiDar ∼ 26K 1 Mapillary PSD [15]…”
Is Pseudo-Lidar needed for Monocular 3D Object detection? (2021, methodology): “When training PackNet [16], the depth network of PL, we use a batch size of 4 and a learning rate of 5 × 10 − 5 with input resolution of 640 × 480 .”