Rendering synthetic objects into legacy photographs

By Kevin Karsch, Varsha Hedau, D.A. Forsyth, Derek Hoiem. 2011

370 citations recorded by the source corpus.

We propose a method to realistically insert synthetic objects into existing photographs without requiring access to the scene or any additional scene measurements. With a single image and a small amount of annotation, our method creates a physical model of the scene that is suitable for realistically rendering synthetic objects with diffuse, specular, and even glowing materials while accounting for lighting interactions between the objects and the scene. We demonstrate in a user study that synthetic images produced by our method are confusable with real scenes, even for people who believe they are good at telling the difference. Further, our study shows that our method is competitive with other insertion methods while requiring less scene information. We also collected new illumination and reflectance datasets; renderings produced by our system compare well to ground truth. Our system has applications in the movie and gaming industry, as well as home decorating and user content creation, among others.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Synthetic Data for Text Localisation in Natural Images (2016, methodology): “[25] develop an impressive semi-automatic method to render objects with correct lighting and perspective; they infer the actual size of objects based on the technique of Criminisi et al.”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “The existing approaches to sythesizing datasets focus largely on ensuring global consistency and realism [5, 12, 22, 50].”
Gibson Env: Real-World Perception for Embodied Agents (2018, background): “This can potentially be solved by integrating our approach with synthetic objects [17, 47].”
3DNN: 3D Nearest Neighbor (2014, methodology): “In addition, the high-ﬁdelity rendering approach of Karsch et al. (2011) produces photorealistic results with accurate lighting effects (reﬂections, scattering, shadows, etc.).”
SAM-Adapter: Adapting Segment Anything in Underperformed Scenes (2023, background): “They offer hints on light source direction and scene illumination that can aid scene comprehension [23, 25].”
Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal (2017, background): “, shadow detection) provides adequate clues about the light sources [25], illumination conditions [38, 39, 40], object shapes [37] and geometry information [19, 20].”
3D-Aware Scene Manipulation via Inverse Graphics (2018, methodology): “Our work is also inspired by previous semi-automatic 3D editing systems [Karsch et al., 2011, Chen et al., 2013, Kholgade et al., 2014].”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, background): “It is one of the core problems in computer vision, with a wide range of applications in gaming, AR/VR, and robotics [12, 14, 40, 43].”
Understanding Bayesian Rooms Using Composite 3D Object Models (2013, background): “Further, we show how to exploit contextual relationships to detect more objects, by, for example, proposing chairs around and underneath tables.”
Intrinsic Scene Properties from a Single RGB-D Image (2013, background): “[16] have attempted to recover the reflectance and illumination of a scene, but assume known geometry and multiple images, or a user annotation of geometry and illumination, respectively.”
OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Previous indoor scene lighting estimation methods only predict shading (which entangles geometry and lighting) [28], require RGBD inputs [4], or rely on hand-crafted heuristics [24, 25].”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Previous indoor scene lighting estimation methods only predict shading (which entangles geometry and lighting) [34], require RGBD inputs [8], or rely on hand-crafted heuristics [29, 30].”
A Multi-Task Mean Teacher for Semi-Supervised Shadow Detection (2020, background): “As a common phenomenon in our daily life, shadows in natural images have hints for extracting the scene geometry [29, 17], light direction [22], camera location and its parameters [16], and beneﬁt different high-level image understanding tasks, e.g., image segmentation [4], object detection [2],…”
Learning to predict indoor illumination from a single image (2017, methodology): “Karsch et al. [2011] estimate scene illumination from single images, but rely on user input to annotate geometry and initial lighting, which is then refined using a rendering-based optimization.”