Generative Hierarchical Features from Synthesizing Images

By Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, Bolei Zhou. 2020

126 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have recently advanced image synthesis by learning the underlying distribution of the observed data. However, how the features learned from solving the task of image generation are applicable to other vision tasks remains seldom explored. In this work, we show that learning to synthesize images can bring remarkable hierarchical visual features that are generalizable across a wide range of applications. Specifically, we consider the pre-trained StyleGAN generator as a learned loss function and utilize its layer-wise representation to train a novel hierarchical encoder. The visual feature produced by our encoder, termed as Generative Hierarchical Feature (GH-Feat), has strong transferability to both generative and discriminative tasks, including image editing, image harmonization, image classification, face verification, landmark detection, and layout prediction. Extensive qualitative and quantitative experimental results demonstrate the appealing performance of GH-Feat.1

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “In terms of sample quality, GANs currently achieve superior performance over VAEs and NFs, but they can be difficult to invert even with additional optimization [48, 100, 7].”
StyleGAN-NADA (2021, background): “These range from image enhancement [18,50], editing [13,36] and recently even discriminative tasks such as classification and regression [27, 48].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, background): “[33] shows that style space can be used for a variety of discriminative and generative tasks.”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, background): “2 [46] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou.”
Low-Rank Subspaces in GANs (2021, methodology): “What surprises us is that our method is extremely robust to the mask (or to the region), which further enables our method could be easily used even if we do not have a segmentation model contrary to the previous works [32, 2, 38, 41].”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, background): “pSp [26] and GHFeat [41] proposed to embed latent codes in a hierarchical manner.”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Examples include tasks like semantic image segmentation [3, 10, 36, 37, 54, 66], optical flow estimation [14, 32, 63], human motion understanding [19, 29, 38, 67], and other dense prediction tasks [3, 70].”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “They also require GAN inversion, which can be inefficient and inaccurate for various datasets (Huh et al., 2020; Karras et al., 2020b; Bau et al., 2019b; Xu et al., 2021).”
LARGE: Latent-Based Regression through GAN Semantics (2021, methodology): “Lastly, GHFeat [73] demonstrated that style-space ( S ) codes extracted using a GAN Inversion encoder are useful features for discriminative tasks.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “also require GAN inversion, which can be hard for certain datasets [24, 32, 8, 64].”
High-fidelity GAN Inversion with Padding Space (2022, methodology): “Recent studies [23, 45] have pointed out that, the constant padding acts as an inductive bias through exposing the spatial location to the generator [45], and hence causes the “texturesticking” problem [23].”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, unknown): “Investigation into StyleSpace [42, 43] studies the style parameters of the channels to determine various properties and editing abilities of StyleGAN.”
One-Shot Generative Domain Adaptation (2021, methodology): “It demonstrates the effectiveness of our method on the general few-shot settings.”
Data-Efficient Instance Generation from Instance Discrimination (2021, background): “The rationale behind self-supervised learning is to set up various pretext tasks with supervisory-free labels [14, 5, 41, 48, 13, 32, 34, 42, 34, 15, 31, 35].”
GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution (2020, result): “This modification is also seen in previous works [11, 36, 46].”