Texture Fields: Learning Texture Representations in Function Space

By Michael Oechsle, Lars M. Mescheder, Michael Niemeyer, Thilo Strauss, Andreas Geiger. 2019

336 citations recorded by the source corpus.

In recent years, substantial progress has been achieved in learning-based reconstruction of 3D objects. At the same time, generative models were proposed that can generate highly realistic images. However, despite this success in these closely related tasks, texture reconstruction of 3D objects has received little attention from the research community and state-of-the-art methods are either limited to comparably low resolution or constrained experimental setups. A major reason for these limitations is that common representations of texture are inefficient or hard to interface for modern deep learning techniques. In this paper, we propose Texture Fields, a novel texture representation which is based on regressing a continuous 3D function parameterized with a neural network. Our approach circumvents limiting factors like shape discretization and parameterization, as the proposed texture representation is independent of the shape representation of the 3D object. We show that Texture Fields are able to represent high frequency texture and naturally blend with modern deep learning techniques. Experimentally, we find that Texture Fields compare favorably to state-of-the-art methods for conditional texture reconstruction of 3D objects and enable learning of probabilistic generative models for texturing unseen 3D models. We believe that Texture Fields will become an important building block for the next generation of generative 3D models.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (2020, neutral): “Coordinate-based MLPs have been used to represent images [28, 38] (referred to as “compositional pattern producing networks”), volume density [27], occupancy [24], and signed distance [32], and have achieved state-of-the-art results across a variety of tasks such as shape representation [9, 10, 12, ”
Pixel-aligned Volumetric Avatars (2021, background): “TextureFields [20] learn to transfer textures from an exemplar image to a source mesh to allow novel view synthesis.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[112] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.”
NeRV: Neural Representations for Videos (2021, methodology): “It has been widely applied in many 3D vision tasks, such as 3D shapes [9, 10], 3D scenes [11–14], and appearance of the 3D structure [4, 15, 16].”
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis (2020, background): “Implicit Representations: Recently, implicit representations of 3D geometry have gained popularity in learning-based 3D reconstruction [10, 15, 35, 42, 45, 48, 59].”
Implicit Neural Representations with Periodic Activation Functions (2020, neutral): “In addition to representing shape, some of these models have been extended to also encode object appearance [3, 5, 10, 14, 15], which can be trained using (multiview) 2D image data using neural rendering [16].”
Convolutional Occupancy Networks (2020, methodology): “Implicit models have been adopted for learning implicit representations from images [23, 24, 29, 41], for encoding texture information [30], for 4D reconstruction [28] as well as for primitive-based reconstruction [10, 11, 15, 32].”
Shape As Points: A Differentiable Poisson Solver (2021, background): “Shape representations are central to many of the recent advancements in 3D computer vision and computer graphics, ranging from neural rendering [41,45,48,55,58] to shape reconstruction [10,26,40, 47, 50, 52, 70].”
DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing (2019, methodology): “Recently, it has been shown that texture can also be represented as an implicit function parameterized with a neural network [33].”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “Neural implicit representations (NIR) have been widely adopted for learning shape and appearance of objects [4,10, 14, 20, 45, 50, 52, 56, 57], local parts [17, 18], and full 3D scenes [7, 11, 29, 55] from explicit 3D supervisions.”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “The core idea is to use a multilayer perceptron (MLPs) to implicitly model the occupancy [41, 42], signed distance functions [53, 2], object appearance [51], volumetric density [16, 36, 63, 67, 64, 44] in the 3D space.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “Inspired by [2–4], we further utilize a deep neural network to parameterize the grasping ﬁeld and learn it from data.”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “Following the practice of Texture Field [50] and PIFu [8], we also regard the surface texture as a vector function deﬁned in the space near the surface, which can support texturing of shapes with arbitrary topology and self-occlusion.”
Neural RGB-D Surface Reconstruction (2021, background): “As input the MLP takes a 3D location in the model space and outputs for example, occupancy [7, 47, 52, 55, 58–60], density [48], radiance [48], color [54], or the signed distance to the surface [56,83,84].”