Point-E: A System for Generating 3D Point Clouds from Complex Prompts

By Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, Mark Chen. arXiv.org, 2022

874 citations recorded by the source corpus.

While recent work on text-conditional 3D object generation has shown promising results, the state-of-the-art methods typically require multiple GPU-hours to produce a single sample. This is in stark contrast to state-of-the-art generative image models, which produce samples in a number of seconds or minutes. In this paper, we explore an alternative method for 3D object generation which produces 3D models in only 1-2 minutes on a single GPU. Our method first generates a single synthetic view using a text-to-image diffusion model, and then produces a 3D point cloud using a second diffusion model which conditions on the generated image. While our method still falls short of the state-of-the-art in terms of sample quality, it is one to two orders of magnitude faster to sample from, offering a practical trade-off for some use cases. We release our pre-trained point cloud diffusion models, as well as evaluation code and models, at https://github.com/openai/point-e.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/1b8a734dd28a9d766a5d3dbc0871e76b6a452b65

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Structured 3D Latents for Scalable and Versatile 3D Generation (2024, positive): “Following Point-E [22], we prepare the point clouds by sampling 4,000 points from unprojected multiview depth maps using the farthest point sampling technique.”
Recent advances in 3D Gaussian splatting (2024, positive): “In addition to 2D SDS, GSGEN [158] introduces a 3D SDS loss based on Point-E [159], a text-to-point-cloud diffusion model, to mitigate the multi-face or Janus problem.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “We compare our SV3D-guided 3D generations with several prior methods including Point-E [33], Shap-E [16], One-2-3-45++ [23], Dream-Gaussian [49], SyncDreamer [26], EscherNet [20], Free3D [62], and Stable Zero123 [48].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Several methods use 3D data for supervised training of text-to-3D models [9, 51, 5]; however this direction remains challenging due to the lack of large-scale aligned datasets of text and 3D. Alternative approaches use 2D vision-language models like CLIP [57] to create 3D content by formulating the…”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “Point-E [46] and Shap-E [30] tend to produce incompleted meshes.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, methodology): “With the recent explosion of generative image models [47, 40, 48, 17, 52, 53, 70, 15], there has been increasing interest in training similar generative models for other modalities such as audio [44, 10, 5, 31, 2, 25], video [24, 57, 23], and 3D assets [26, 28, 45, 41].”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “We compare Magic123 to recent methods (Point-E [58], ShapeE [34], 3DFuse [77], RealFusion [51], and Zero-1-to-3 [46]) for generating 3D objects from a single unposed image (the leftmost column).”
LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “The models of Point-E and Shap-E contain hundreds of millions of learnable parameters and are trained with several million 3D assets (unknown data source and unknown computational cost from their papers).”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “Early promising diffusion-based results [2, 16, 49, 62, 101, 116] have mainly been restricted by lacking availability of useful real-world multi-view training data.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “In addition to these methods, OpenAI trains a 3D native diffusion model Point-E [52], which uses several million internal 3D models to generate point clouds.”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, background): “Some works attempt to train text-conditioned diffusion models on 3D representations such as point clouds and volumes [1,5,9,16,26,33,35,53,58,61].”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Theoretically, 3D native methods (Jun & Nichol, 2023; Nichol et al., 2022; Gupta et al., 2023) exhibit the potential to generate 3D-consistent assets within seconds, albeit at the cost of requiring extensive training on large-scale 3D datasets.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “Imagine a scenario where one can inter-actively create and edit 3D content through simple verbal commands [22,30], bypassing the need for specialized software, or can instruct a robot to manipulate objects using nat-ural language [10].”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “Another line of research [26, 51, 59, 55] projects the input image into the latent space of the pretrained 3D-aware generative networks.”
MVDream: Multi-view Diffusion for 3D Generation (2023, background): “It’s pertinent to note ongoing research endeavors to reconstruct object shapes directly from monocular image inputs (Wu et al., 2023; Nichol et al., 2022; Jun & Nichol, 2023), aligning with the increasing stability of image generation techniques.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “In addition to that, there are also generation methods that utilize point clouds [41, 48, 53, 74] and meshes [37] as 3D representations.”
3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, positive): “Similarly, for point-to-point generation, we use Point-E (Nichol et al., 2022) as the pretrained model, to which we add a point cloud condition input.”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, background): “Recently, diffusion-based techniques have been adapted for point cloud generation [44, 67, 68], showing competitive performance in shape generation.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “Point-E [45] and Shap-E [26] are 3D generative models trained on a large internal OpenAI 3D dataset, both of which are able to convert a single-view image into a point cloud or an implicit representation.”