Neural Style Transfer via Meta Networks

By Falong Shen, Shuicheng Yan, Gang Zeng. 2018

134 citations recorded by the source corpus.

In this paper we propose a noval method to generate the specified network parameters through one feed-forward propagation in the meta networks for neural style transfer. Recent works on style transfer typically need to train image transformation networks for every new style, and the style is encoded in the network parameters by enormous iterations of stochastic gradient descent, which lacks the generalization ability to new style in the inference stage. To tackle these issues, we build a meta network which takes in the style image and generates a corresponding image transformation network directly. Compared with optimization-based methods for every style, our meta networks can handle an arbitrary new style within 19 milliseconds on one modern GPU card. The fast image transformation network generated by our meta network is only 449 KB, which is capable of real-time running on a mobile device. We also investigate the manifold of the style transfer networks by operating the hidden features from meta networks. Experiments have well validated the effectiveness of our method. Code and trained models will be released.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xiaogang Wang
3. Luc Van Gool
4. Yi Yang
5. Bernt Schiele
6. Trevor Darrell
7. Ming-Hsuan Yang
8. Hongsheng Li

In Generative Models

1. Greg Mori
2. Mengyao Zhai
3. Lei Chen
4. Amos J. Storkey
5. Antreas Antoniou
6. Hongsheng Li
7. Xiaogang Wang
8. Lu Sheng

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “Image I Object detection (face [38], [181], [182], facial point [183], pedestrian [184], general [31], [185], [186], [187], [188]) Image segmentation [101], [189], Super resolution [190], Style transfer [191], Coarse-to-fine classification [192]”
• Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “The gram matrix is adopted on the level of feature map to encode such semantically-irrelevant information following prior works [8, 13, 23, 24, 31].”
• Meta-Consolidation for Continual Learning (2020, background): “This opened up an interesting research direction, in exploiting the meta-manifold of model parameters for multi-task learning [66], neural architecture search [8], zero-shot learning [57, 81] and style transfer [72] in recent years.”
• GAN Memory with No Forgetting (2020, background): “Style-transfer techniques An extensive literature [23, 71, 79, 13, 62, 38] has explored how one can manipulate the style of an image (e.”
• Meta-Learning in Neural Networks: A Survey (2020, background): “Meta-learning has also accelerated style transfer by learning a FFM to solve the corresponding optimization problem [216].”
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis (2019, background): “inputs. Ha et al. [7] proposed HyperNetworks, where a hyper-network is used to generate weights for another network. This idea has been applied to different applications such as neural style transfer [27], superresolution [15, 12], image segmentation [8, 30], motion prediction [32] and tracking [19]”
• Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (2021, background): “and even using different training paradigms [45, 56].”
• Hyper-LifelongGAN: Scalable Lifelong Learning for Image Conditioned Generation (2021, methodology): “This idea has been applied to applications in different research fields such as few shot learning [27, 33], image segmentation [2, 36] and generative models [28, 18, 16], which is the focus of our paper.”
• StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, methodology): “Recently, a number of works [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] approximated the iterative back-propagation procedure as the feed-forward networks for real-time style transfer.”
• Neural Style Transfer: A Review (2017, background): “In [58], Shen et al.”
• Dynamic Instance Normalization for Arbitrary Style Transfer (2019, methodology): “This point is also partly observed in (Shen, Yan, and Zeng 2018).”