Semantic Pyramid for Image Generation

By Assaf Shocher, Yossi Gandelsman, Inbar Mosseri, Michal Yarom, Michal Irani, William T. Freeman, Tali Dekel. 2020

58 citations recorded by the source corpus.

We present a novel GAN-based model that utilizes the space of deep features learned by a pre-trained classification model. Inspired by classical image pyramid representations, we construct our model as a Semantic Generation Pyramid -- a hierarchical framework which leverages the continuum of semantic information encapsulated in such deep features; this ranges from low level information contained in fine features to high level, semantic information contained in deeper features. More specifically, given a set of features extracted from a reference image, our model generates diverse image samples, each with matching features at each semantic level of the classification model. We demonstrate that our model results in a versatile and flexible framework that can be used in various classic and novel image generation tasks. These include: generating images with a controllable extent of semantic similarity to a reference image, and different manipulation tasks such as semantically-controlled inpainting and compositing; all achieved with the same model, with no further training.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Projected GANs Converge Faster (2021, neutral): “Pretrained models can be used as a guiding mechanism to disentangle causal generative factors [54], for text-driven image manipulation [44], matching the generator activations to inverted classifiers [19, 56], or to generate images via gradient ascent in the latent space of a generator [41].”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, methodology): “In Shocher et al. (2020), a hierarchical LSGAN generator is trained with a class-conditional discriminator (Zhang et al., 2019a).”
Network-to-Network Translation with Conditional Invertible Neural Networks (2020, methodology): “To address this, [48] learns an autoregressive model conditioned on speciﬁc layers of pre-trained models and [63] a GAN based decoder conditioned on a feature pyramid of a pre-trained classiﬁer.”
Deep ViT Features as Dense Visual Descriptors (2021, background): “Features of pre-trained CNNs are a cornerstone for a plethora of vision tasks from object detection and segmentation [18, 6], to image generation [43].”
Understanding invariance via feedforward inversion of discriminatively trained classifiers (2021, methodology): “This view is generally supported by methods that have been developed to invert internal representations and recover the visual information that is retained by the representation in a given layer (Mahendran and Vedaldi, 2015; Dosovitskiy and Brox, 2016a;b; Zhang et al., 2016; Shocher et al., 2020).”
Using latent space regression to analyze and leverage compositionality in GANs (2021, methodology): “For image synthesis, Mokady et al. (2019) and Press et al. (2020) use an autoencoder to encode, disentangle, and swap properties between two sets of images, while Shocher et al. (2020) mixes images in deep feature space while training the generator.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Semantic Generation Pyramid [58] achieves semantic image composition by combining the deep features of the foreground image and background image, but the result is not very similar to the foreground image.”
Generative Hierarchical Features from Synthesizing Images (2020, background): “[48] discover the potential of discriminative features in image generation, but the transferability of these features are still not fully verified.”
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (2021, background): “Depending on the input type, such as class-label [3, 43, 54], segmentation mask [31, 47], feature from classifier [28, 38], and image [17, 26], various conditional generative models are available.”
Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs (2020, background): “Recently, [54] introduced a GAN-based approach that utilizes features of a pre-trained classifier as a semantic pyramid for image generation.”
Image-to-Image Translation: Methods and Applications (2021, background): “of objects (image composition) [186] with or without makeup [187].”