PolyGen: An Autoregressive Generative Model of 3D Meshes

By Charlie Nash, Yaroslav Ganin, Ali Eslami, Peter W. Battaglia. International Conference on Machine Learning, 2020

373 citations recorded by the source corpus.

Polygon meshes are an efficient representation of 3D geometry, and are of central importance in computer graphics, robotics and games development. Existing learning-based approaches have avoided the challenges of working with 3D meshes, instead using alternative object representations that are more compatible with neural architectures and training approaches. We present an approach which models the mesh directly, predicting mesh vertices and faces sequentially using a Transformer-based architecture. Our model can condition on a range of inputs, including object classes, voxels, and images, and because the model is probabilistic it can produce samples that capture uncertainty in ambiguous scenarios. We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task. We also evaluate the conditional models on surface reconstruction metrics against alternative methods, and demonstrate competitive performance despite not training directly on this task.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/84fae4c57e9f65459cf404866f83ff0b70fd7b75

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Graph neural networks in particle physics (2020, neutral): “For example, [80] and [81] are autogressive graph generators, which use a GN or Transformer to predict whether a new vertex should be added to a graph (by”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, background): “3D Models Representation 3D-GAN [Wu et al. 2016] GAN Voxels l-GAN [Achlioptas et al. 2018] GAN Point Clouds IM-GAN [Chen and Zhang 2019] GAN Fields PointFlow [Yang et al. 2019] NF Point Clouds GenVoxelNet [Xie et al. 2020] EBM Voxels PointGrow [Sun et al. 2020] AR Point Clouds PolyGen [Nash et al. 2”
Accelerating 3D deep learning with PyTorch3D (2019, background): “Predicting depth [10, 9], or 3D shape [8, 11, 59, 41] can lift 2D images into 3D. Generative models [61, 62, 39] might one day aid artists in authoring 3D content.”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “In the 3D domain, autoregressive models have been used to learn the distribution of point clouds [60, 69] and meshes [47].”
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, background): “Recently, there has been tremendous progress in 3D shape generating in different data formats such as point cloud [3,34,59], voxel [58], implicit representation [8,37,44] and mesh [39].”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, methodology): “For Polygen [43], we re-implement it following the details in their paper.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “By combing 2D image encoder and 3D generators, they generates 3D data in various representations, including 3D voxels [18, 79, 10, 81, 80, 85], point clouds [15, 88, 19, 1, 44, 90], polygon meshes [30, 73, 77, 51], and parametric models [54, 94, 95].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, background): “…directly from 3D synthetic data or real scans and generate various 3D representations such as point clouds [1, 15, 41, 47, 83], 3D voxels [9, 59, 74, 75], polygon meshes [16, 17, 26, 32, 37, 46, 68], parametric models [21], and implicit fields [8, 14, 19, 25, 30, 42, 48, 73, 78, 82, 84, 86, 87].”
SceneFormer: Indoor Scene Generation with Transformers (2020, background): “Transformers [31] perform well on a variety of natural language processing tasks by treating a sentence as a sequence of words, and recently on images [20] and 3D meshes [23] as well.”
DeepCAD: A Deep Generative Network for Computer-Aided Design Models (2021, methodology): “For instance, PolyGen [29] leverages Transformer to generate vertices and faces of 3D meshes in a sequential way.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, neutral): “The output of the last tTPF layer is then reshaped into three axis-aligned feature planes of size 256 × 256 × 32 .”
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (2021, positive): “Surface-based Methods directly predict triangular meshes and have achieved impressive results for reconstructing and synthesizing simpler shapes [5, 7, 23, 41, 54].”
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation (2020, methodology): “Inspired by works such as PixelCNN [16] and PolyGen [15], which discretize continuous signals, we first quantize the input coordinates to 8-bits.”
Learning Mesh-Based Simulation with Graph Networks (2020, background): “While meshes are sometimes used for learned geometry processing [9] and generative models of shapes [15, 29], most work on predicting high-dimensional physical systems focuses on grids, owing to the popularity and hardware support for CNN architectures [19].”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, neutral): “Polygen [Nash et al. 2020] and MeshGPT [Siddiqui et al. 2024] take a different approach by natively representing meshes through points and surface sequences.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “Following PolyGen (Nash et al., 2020), we use an addition of three learned embeddings as the embedding of each vertex token: Coordinate Embedding , indicating whether the token represents x or y coordinate; Position Embedding , representing which vertex the token belongs to; Value Embedding , expres”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, background): “PolyGen [51], SurfGen [44], and GET3D [22] generate meshes with arbitrary topology.”