Positional Encoding as Spatial Inductive Bias in GANs

By Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, Chen Change Loy. 2020

96 citations recorded by the source corpus.

SinGAN shows impressive capability in learning internal patch distribution despite its limited effective receptive field. We are interested in knowing how such a translationinvariant convolutional generator could capture the global structure with just a spatially i.i.d. input. In this work, taking SinGAN and StyleGAN2 as examples, we show that such capability, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators. Such positional encoding is indispensable for generating images with high fidelity. The same phenomenon is observed in other generative architectures such as DCGAN and PGGAN. We further show that zero padding leads to an unbalanced spatial bias with a vague relation between locations. To offer a better spatial inductive bias, we investigate alternative positional encodings and analyze their effects. Based on a more flexible positional encoding explicitly, we propose a new multi-scale training strategy and demonstrate its effectiveness in the state-of-the-art unconditional generator StyleGAN2. Besides, the explicit spatial inductive bias substantially improves SinGAN for more versatile image manipulation. 1

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs (2021, background): “A more recent study explored zero padding in generative models in a much deeper and systematic way and showed that CNNs use it to produce more robust spatial structure [33].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “The synthesis network G of StyleGAN3 starts from a spatial map defined by Fourier features [Tancik et al. 2020; Xu et al. 2021].”
Alias-Free Generative Adversarial Networks (2021, methodology): “known to leak absolute image coordinates into the internal representations [27, 34, 63].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, background): “When a short video clip is encoded, the zero paddings in the temporal dimension also get encoded and affect the output tokens, causing an unbalanced effects to tokens at different temporal position [24,33,80,3].”
High-fidelity GAN Inversion with Padding Space (2022, background): “A similar phenomenon is also observed in generative models [44].”
StyleSwin: Transformer-based GAN for High-resolution Image Generation (2021, methodology): “In view of this, we propose to introduce sinusoidal position encoding [7, 57, 61] (SPE) on each scale, as shown in Figure 2(b).”
Exploring CLIP for Assessing the Look and Feel of Images (2022, methodology): “Unlike convolutional models that implicitly capture positional information (Xu et al. 2021), Transformer relies heavily on the positional embedding to provide such clue.”
StyTr2: Image Style Transfer with Transformers (2021, unknown): “[59] and [23] further include positional encoding in CNN-based models as spatial inductive.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Second, the connections between convolutions and stationary processes have recently been highlighted by Xu et al. (2020) as enforcing PE.”