On Self Modulation for Generative Adversarial Networks

By Ting-Li Chen, Mario Lucic, Neil Houlsby, Sylvain Gelly. 2018

111 citations recorded by the source corpus.

Training Generative Adversarial Networks (GANs) is notoriously challenging. We propose and study an architectural modification, self-modulation, which improves GAN performance across different data sets, architectures, losses, regularizers, and hyperparameter settings. Intuitively, self-modulation allows the intermediate feature maps of a generator to change as a function of the input noise vector. While reminiscent of other conditioning techniques, it requires no labeled data. In a large-scale empirical study we observe a relative decrease of $5\%-35\%$ in FID. Furthermore, all else being equal, adding this modification to the generator leads to improved performance in $124/144$ ($86\%$) of the studied settings. Self-modulation is a simple architectural change that requires no additional parameter tuning, which suggests that it can be applied readily to any GAN.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, methodology): “This technique, similar to conditional batch normalization (De Vries et al., 2017), FiLM (Perez et al., 2018), and selfmodulation (Chen et al., 2019), also yields parameterefficient adaptation of a network; with only 2d parameters per layer.”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “[5], it is becoming clear that the traditional GAN generator architecture is in every way inferior to a style-based design.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “C V ] 1 Recently, the performance of unconditional GANs has been fundamentally advanced, chiefly owing to the success of modulation approaches (Chen et al., 2019; Karras et al., 2019a;b) with learned style representations produced by a latent vector.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “In contrast to [5] and in alignment with [6], we find it beneficial not to use a hierarchical latent space, but to directly feed the same input vector z to BatchNorm at every layer in the generator.”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Both were first used in the style transfer task and later adopted in various vision tasks [3, 8, 10, 20, 26, 36, 39, 42, 49, 54].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “This formulation is also known as self-modulation [6].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “As a result, many techniques to stabilize GAN training have been proposed [4, 5, 6, 7, 8, 9, 10].”
High-Fidelity Image Generation With Fewer Labels (2019, neutral): “While this performance is still considerably worse than that of B IG GAN this result is the current state-of-the-art in unsupervised image generation (Chen et al. (2019b) report an FID of 33 for unsupervised generation).”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Fine-tuning [66] pretrained LMs is the standard practice for learning downstream tasks, which may involve simple architecture modiﬁ-1 cations, such as adding linear layers [67, 68] or freezing layers [18, 69, 70].”
Improved Transformer for High-Resolution GANs (2021, methodology): “First, as shown in [8], self-modulation stabilizes the generator towards favorable conditioning values and also appears to improve mode coverage.”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, background): “Self-modulation [4] shows GANs benefit from self-modulation layers in the generator.”
MaxViT: Multi-Axis Vision Transformer (2022, methodology): “Similar to [98], we apply a cross-attention layer before the MaxViT block as a memory-efficient form of self-modulation in every stage, which has been shown to stabilize GAN training and also improve mode coverage [8,98].”
Dual Contrastive Loss and Attention for GANs (2021, background): “As attention models outperform others in various computer vision tasks, researchers were quick to incorporate them into unconditional image generation [10, 98, 65, 5], semantic-based image generation [56, 75], and text-guided image manipulation models [48, 66].”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, background): “Network architectures can play a vital role in both improving training stability [7] and controllability of generated images [24, 38].”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, methodology): “Inspired by [8, 26, 27], we design a style manipulation module to endow our framework with pluralistic generation.”
Semantic Bottleneck Scene Generation (2019, background): “Training GANs is notoriously hard and recent efforts focused on improving neural architectures [21, 44, 9], loss functions [1], regularization [15, 30], large-scale training [7], self-supervision [10], and sampling [7, 4].”
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, methodology): “A similar approach proposed by Chen et al. [8] shows that this network design also achieves more training stability.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “[266] proposed a more general self-modulation based on CBN, where the afﬁne parameters can also be generated by the generators own input or provided by external information.”