RealFusion 360° Reconstruction of Any Object from a Single Image

By Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Andrea Vedaldi. Computer Vision and Pattern Recognition, 2023

376 citations recorded by the source corpus.

We consider the problem of reconstructing a full 360° photographic model of an object from a single image of it. We do so by fitting a neural radiance field to the image, but find this problem to be severely ill-posed. We thus take an off-the-self conditional image generator based on diffusion and engineer a prompt that encourages it to “dream up” novel views of the object. Using the recent DreamFusion method, we fuse the given input view, the conditional prior, and other regularizers into a final, consistent reconstruction. We demonstrate state-of-the-art reconstruction results on benchmark images when compared to prior methods for monocular 3D reconstruction of objects. Qualitatively, our reconstructions provide a faithful match of the input view and a plausible extrapolation of its appearance and 3D shape, including to the side of the object not visible in the image.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/84cce9b8aea35e4fa38eef63da439573f21c0728

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “…generative models (e.g. DALL-E [54] and Stable Diffusion [56]) with impressive and high-quality image generation, a large number of studies [7, 33, 40, 50] incorporate 2D diffusion and CLIP priors into per-shape optimization via various score distillation sampling strategy [50, 62, 66], or with…”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “Concurrent works such as RealFusion [21] and SparseFusion [54] explore grounding by providing one or few input images, where the unseen parts are hallucinated.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “By applying a NeRF model to distill the Stable Diffusion model [49, 54], RealFusion [43] shows strong multiview consistency producing more reconstructed points but is unable to produce visually plausible images as shown in Fig.”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, neutral): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “We further use the dataset collected and released by RealFusion [51], consisting of 15 natural images that include bananas, birds, cacti, barbie cakes, cat statues, teapots, microphones, dragon statues, fishes, cherries, and watercolor paintings etc.”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “Recent methods [56, 41, 45, 88, 44] combine large text-to-image diffusion models [65, 67] and neural radiance fields [47] to generate 3D objects without training.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, methodology): “Concurrent approaches [9, 29, 61] proposed similar techniques to perform image-to-3D generation using language-guided priors and textual inversion [14].”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “, 2023a) and average generation time of different image-to-3D methods on a collection of images from previous works (Melas-Kyriazi et al., 2023; Liu et al., 2023a; Tang et al., 2023b) and Internet.”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “We follow RealFusion [43] to report F-score (with a threshold of 0.05) and CLIP similarity, and the runtime on an A100 GPU.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Using text-based priors for single-image-to-3D has also shown promise [29, 30, 31], but requires a complex balancing of the image observation with additional constraints.”
Continuous 3D Perception Model with Persistent State (2025, neutral): “With rapid advances in image and video generative models [57, 61, 62], much of the current research in 3D generation focuses on transferring priors from images and videos to 3D [45, 48, 56, 91].”
Splatter Image: Ultra-Fast Single-View 3D Reconstruction (2023, positive): “Other works build and use a 3D [6, 25] or 2D [7, 22] prior which can be used in an image-conditioned auto-decoding framework.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, positive): “NeRF based SLAM is a relatively under-explored area of research. iMAP [73] and Nice-SLAM [74] offer excellent NeRF based SLAM frameworks which could integrate faster and better quality NeRF models.”
Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “Reconstructing 3D geometry from a single image [13, 26, 33, 38, 40, 42, 45] stands as a fundamental task in computer graphics and 3D computer vision, benefiting a wide range of versatile applications such as novel view synthesis [7, 25, 35], 3D content creation [36, 48], and robotics grasping [29,…”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “Techniques such as RegNeRF [28], which uses geometry loss from depth patches, and SinNeRF [49], RealFusion [22], and NeuralLift [50], which combine depth maps or Score Dis-tillation Sampling during NeRF training, represent significant steps forward.”