Dynamic Instance Normalization for Arbitrary Style Transfer

By Yongcheng Jing, Xinyu Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen. 2019

219 citations recorded by the source corpus.

Prior normalization methods rely on affine transformations to produce arbitrary image style transfers, of which the parameters are computed in a pre-defined way. Such manually-defined nature eventually results in the high-cost and shared encoders for both style and content encoding, making style transfer systems cumbersome to be deployed in resource-constrained environments like on the mobile-terminal side. In this paper, we propose a new and generalized normalization module, termed as Dynamic Instance Normalization (DIN), that allows for flexible and more efficient arbitrary style transfers. Comprising an instance normalization and a dynamic convolution, DIN encodes a style image into learnable convolution parameters, upon which the content image is stylized. Unlike conventional methods that use shared complex encoders to encode content and style, the proposed DIN introduces a sophisticated style encoder, yet comes with a compact and lightweight content encoder for fast inference. Experimental results demonstrate that the proposed approach yields very encouraging results on challenging style patterns and, to our best knowledge, for the first time enables an arbitrary style transfer using MobileNet-based lightweight architecture, leading to a reduction factor of more than twenty in computational cost as compared to existing approaches. Furthermore, the proposed DIN provides flexible support for state-of-the-art convolutional operations, and thus triggers novel functionalities, such as uniform-stroke placement for non-natural images and automatic spatial-stroke control.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Xinchao Wang
2. Dacheng Tao
3. Ling Shao
4. Mingli Song
5. Li Liu
6. Fan Zhu
7. Yiding Yang
8. Errui Ding

Citations in the corpus

• Learning with Recoverable Forgetting (2022, neutral): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
• AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “Similar trade-off between flexibility and capability also exists in [5, 22, 15, 24, 10], where all local feature points of the content image are processed by the same transformation function based on style images.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “…how to compute the affine parameters so as to align the mean and variance between content and style features, dynamic instance normalization (DIN) [244], introduced by Jing et al ., deals with arbitrary style transfer by encoding a style image into learnable convolution parameters, upon which the…”
• Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “In the past few years, multi-task learning has been widely studied in various areas, such as bioinformatics [10, 20], ubiquitous computing [39, 40], natural language processing [4, 38] and computer vision [21, 54, 15, 14].”