CAT3D: Create Anything in 3D with Multi-View Diffusion Models

By Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, Ben Poole. Neural Information Processing Systems, 2024

457 citations recorded by the source corpus.

Advances in 3D reconstruction have enabled high-quality 3D capture, but require a user to collect hundreds to thousands of images to create a 3D scene. We present CAT3D, a method for creating anything in 3D by simulating this real-world capture process with a multi-view diffusion model. Given any number of input images and a set of target novel viewpoints, our model generates highly consistent novel views of a scene. These generated views can be used as input to robust 3D reconstruction techniques to produce 3D representations that can be rendered from any viewpoint in real-time. CAT3D can create entire 3D scenes in as little as one minute, and outperforms existing methods for single image and few-view 3D scene creation. See our project page for results and interactive demos at https://cat3d.github.io .

Subfields: Computer Graphics and Computer-Aided Design, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/4987a76781f299be64ac43419c8b489ca1f4515b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Management Science and Operations Research

Citations in the corpus

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “Methods like [14], [63] use predefined camera trajectories for scene generation, which overlooks the diverse geometry relationships presented in different scenes, resulting in significant occlusions.”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, unknown): “With the recent success of image/video generation models, Re-conFusion [61] and CAT3D [12] started leveraging prior knowledge learned by these models to facilitate sparse-view NVS. Due to the necessity of per-scene optimization, these methods remain inherently slow.”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, background): “Recently, there have been growing interests [1,3,5,12,26,36,39,43,44,51] in scene reconstruction and synthesis from sparse input views ( e.g ., 2 or 3).”
Continuous 3D Perception Model with Persistent State (2025, unknown): “The virtual camera’s intrinsics and extrinsics are represented as a raymap R , a 6-channel image encoding the origin and direction of rays at each pixel [25, 90, 99].”