ZM-Net: Real-time Zero-shot Image Manipulation Network

By Hao Wang, Xiaodan Liang, H. Zhang, Dit-Yan Yeung, Eric P. Xing. 2017

34 citations recorded by the source corpus.

Many problems in image processing and computer vision (e.g. colorization, style transfer) can be posed as 'manipulating' an input image into a corresponding output image given a user-specified guiding signal. A holy-grail solution towards generic image manipulation should be able to efficiently alter an input image with any personalized signals (even signals unseen during training), such as diverse paintings and arbitrary descriptive attributes. However, existing methods are either inefficient to simultaneously process multiple signals (let alone generalize to unseen signals), or unable to handle signals from other modalities. In this paper, we make the first attempt to address the zero-shot image manipulation task. We cast this problem as manipulating an input image according to a parametric model whose key parameters can be conditionally generated from any guiding signal (even unseen ones). To this end, we propose the Zero-shot Manipulation Net (ZM-Net), a fully-differentiable architecture that jointly optimizes an image-transformation network (TNet) and a parameter network (PNet). The PNet learns to generate key transformation parameters for the TNet given any guiding signal while the TNet performs fast zero-shot image manipulation according to both signal-dependent parameters from the PNet and signal-invariant parameters from the TNet itself. Extensive experiments show that our ZM-Net can perform high-quality image manipulation conditioned on different forms of guiding signals (e.g. style images and attributes) in real-time (tens of milliseconds per image) even for unseen signals. Moreover, a large-scale style dataset with over 20,000 style images is also constructed to promote further research.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

GAN Memory with No Forgetting (2020, background): “Style-transfer techniques An extensive literature [23, 71, 79, 13, 62, 38] has explored how one can manipulate the style of an image (e.”
Universal Style Transfer via Feature Transforms (2017, background): “Meanwhile, inspired by [6], two following work [30, 11] turn to learn a general mapping from the style image to style parameters.”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “The existing attempts either tried to adjust the holistic statistics of the content features with that of the style features [17, 18, 19, 20, 21, 22, 23], or non-locally swap the relevant style features in order to match the content features [24, 25, 26, 27, 28, 29, 30, 31].”
Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification (2020, methodology): “For decoders, following (Wang et al. 2017), we use 4 residual blocks with Adaptive Instance Normalization (AdaIN) and 2 upsampling with convolutional layers.”
Arbitrary Style Transfer With Style-Attentional Networks (2018, background): “Several approaches [1, 8, 12, 22, 3, 14, 19, 26, 29] have been developed based on feedforward networks.”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, methodology): “A common practice is employing external style signals to supervise the content modification [12, 25, 31, 29, 3] on a feed-forward network.”
Generative Semantic Manipulation with Contrasting GAN (2017, background): “GANs have shown great success on a variety of image-conditional models such as style transfer [11, 29] and general-purpose image-to-image translation [9].”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, background): “Beyond that, a number of methods have been proposed to address different aspects of style transfer, including quality [14,49,4,47,23], diversity [28,44], photorealism [32], combining several styles in a single model [45,7,3] and generalizing to previously unseen styles [21,29,15,38].”