Spatially Controllable Image Synthesis with Internal Representation Collaging

By Ryohei Suzuki, Masanori Koyama, Takeru Miyato, Taizan Yonetsuji, Huachun Zhu. 2018

43 citations recorded by the source corpus.

We present a novel CNN-based image editing strategy that allows the user to change the semantic information of an image over an arbitrary region by manipulating the feature-space representation of the image in a trained GAN model. We will present two variants of our strategy: (1) spatial conditional batch normalization (sCBN), a type of conditional batch normalization with user-specifiable spatial weight maps, and (2) feature-blending, a method of directly modifying the intermediate features. Our methods can be used to edit both artificial image and real image, and they both can be used together with any GAN with conditional normalization layers. We will demonstrate the power of our method through experiments on various types of GANs trained on different datasets. Code will be available at this https URL.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Rewriting a Deep Generative Model (2020, methodology): “…and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo stylization [25,38,56,54], image-to-image translation [35,86,9,73,55,32], and face editing and…”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
Editing in Style: Uncovering the Local Semantics of GANs (2020, methodology): “Ours Feature blend 32× 32 [26] Feature blend 64× 64 [26] Poisson blend [24]”
Transforming and Projecting Images into Class-conditional Generative Networks (2020, methodology): “By using the ground truth class label, Neural Collage [60] turn a BigGAN into a single-class GAN and adopts single-class image projection [70].”
Low-Rank Subspaces in GANs (2021, methodology): “Consequently, they highly rely on spatial masks [32, 9, 2] to locally control the GAN generation, which is of more practical usage.”
Using latent space regression to analyze and leverage compositionality in GANs (2021, methodology): “Suzuki et al. (2018) use a spatial feature blending approach which mixes properties of target images in the intermediate feature space of a generator.”
Repurposing GANs for One-Shot Semantic Part Segmentation (2021, background): “[47] show that interchanging activations between images can result in interchanging of objects in the output image.”
Style Intervention: How to Achieve Spatial Disentanglement with Style-based Generators? (2020, background): “From another perspective, manipulating feature maps along the spatial dimension [4, 24], e.”