Accelerating 3D deep learning with PyTorch3D

By Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan-Yen Lo, Justin Johnson, Georgia Gkioxari. 2019

1,177 citations recorded by the source corpus.

1. Accelerating 3D Deep Learning with PyTorch3D, arXiv 2007.08501 2. Mesh R-CNN, ICCV 2019 3. SynSin: End-to-end View Synthesis from a Single Image, CVPR 2020 4. Fast Differentiable Raycasting for Neural Rendering using Sphere-based Representations, arXiv 2004.07484

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

In ML Systems & Hardware

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “Weakly supervised approaches use category-specific priors via 3D shape templates [17, 25, 29] and pose [42] or learn via 2D silhouettes and re-projection on posed views [6,26,34,49].”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, methodology): “For each mask M , we estimate R ∈ SO (3) and T ∈ R 3 such that the following silhouette loss is minimized where DR ( · ) is a differentiable renderer implemented in PyTorch3D [54].”
ICON: Implicit Clothed humans Obtained from Normals (2021, methodology): “Under a weak-perspective camera model, with scale s ∈ R and translation t ∈ R 3 , we use the PyTorch3D [51] dif-3 We train the normal networks, G N , with the following loss: where back } , is a loss (L1) between ground-truth and predicted normals (the two G N in Fig.”
Modular primitives for high-performance differentiable rendering (2020, methodology): “PyTorch3D [Ravi et al. 2020] scales mostly with the total area of geometry, occluded or not, whereas our method is not slowed down by hidden geometry.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “Then, we randomly select the constructed point cloud of the video frames and render it along the estimated camera trajectory using Pytorch3D [71].”
Advances in Neural Rendering (2021, methodology): “This has been implemented, for example, in [LLCL19,RRN ∗ 20].”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, methodology): “The rest is the same as in MLP. Finally, the Mesh baseline uses the soft-rasterization of [6] as implemented in PyTorch3D [45] with the top-k face accumulation.”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, methodology): “Here, several methods explore novel representations [35, 36, 37, 38], network architectures [39, 40] and differentiable rendering techniques [41, 42, 43, 44, 45] to predict the 3D shapes and textures of objects from images with or without 3D supervision.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, methodology): “We differentiably render the resulting textured mesh using PyTorch3D [49].”
AutoFlow: Learning a Better Training Set for Optical Flow (2021, background): “Differentiable rendering [29, 37] enables gradients to be passed to rendering parameters, which, however, do not directly relate to the scene distribution hyperparameters.”
MVTN: Multi-View Transformation Network for 3D Shape Recognition (2020, methodology): “In this paper, MVTN takes advantage of differentiable rendering [41, 53, 66].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, neutral): “G. Figure 13 shows randomly selected scenes with synthetically rendered robot masks using PyTorch3D [44], qualitatively demonstrating the high accuracy of our filtered camera-to-robot base calibration”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, methodology): “It is based on a differentiable renderer [19] implemented in PyTorch3D [28].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “We implement mesh rasterization and fusion with Pytorch3D [59].”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, methodology): “Our models are implemented with Py-Torch [54], and we use PyTorch3D [34] to perform differentiable conversion between rotation representations.”