Rewriting a Deep Generative Model

By David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba. 2020

162 citations recorded by the source corpus.

A deep generative model such as a GAN learns to model a rich set of semantic and physical rules about the target distribution, but up to now, it has been obscure how such rules are encoded in the network, or how a rule could be changed. In this paper, we introduce a new problem setting: manipulation of specific rules encoded by a deep generative model. To address the problem, we propose a formulation in which the desired rule is changed by manipulating a layer of a deep network as a linear associative memory. We derive an algorithm for modifying one entry of the associative memory, and we demonstrate that several interesting structural rules can be located and modified within the layers of state-of-the-art generative models. We present a user interface to enable users to interactively change the rules of a generative model to achieve desired effects, and we show several proof-of-concept applications. Finally, results on multiple datasets demonstrate the advantage of our method against standard fine-tuning methods and edit transfer algorithms.

Subfields: Artificial Intelligence, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Signal Processing

Citations in the corpus

GAN Inversion: A Survey (2021, background): “7 7 7 7 7 3 [94] [36] GANRewriting [98] 2020, ECCV O.”
Locating and Editing Factual Associations in GPT (2022, methodology): “Compared to previous ﬁne-tuning (Zhu et al., 2020), interpretability-based (Dai et al., 2022), and meta-learning (Mitchell et al., 2021; De Cao et al., 2021) methods, ROME achieves good generalization and speciﬁcity simultaneously, whereas previous approaches sacriﬁce one or the other.”
Alias-Free Generative Adversarial Networks (2021, background): “While several ways of controlling the generative process have been found [8, 28, 10, 41, 24, 2, 7, 46, 6], the foundations of the synthesis process remain only partially understood.”
Post-hoc Concept Bottleneck Models (2022, background): “Concepts Using human concepts to interpret model behavior has been drawing increasing interest (Kim et al., 2018; Bau et al., 2017; 2020).”
EditGAN: High-Precision Semantic Image Editing (2021, background): “For example, [25, 28] essentially specialize the generator network for certain images at test time to aid image embedding or “rewrite” the network to achieve desired semantic changes in output.”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Similarly, model editing [5, 70, 19, 69, 45, 38, 40, 39] aims to modify specific model weights based on users’ instructions to incorporate new computational rules or new visual effects.”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “To quantify undesired changes (e.g. change of identity), we follow Bau et al. (2020) to compute masked LPIPS (Zhang et al., 2018).”
Navigating the GAN Parameter Space for Semantic Image Editing (2020, background): “In a nutshell, [1] solves the optimization problem that seeks the shift in the generator’s parameter space, such that the modified generator would produce the images after a needed editing operation.”
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs (2020, background): “One way is to treat pretrained GANs as strong image priors, which can be achieved by either exploiting GANs as a component of their networks [22, 5, 6, 17, 3] or directly editing the latent code to find feasible solutions [1, 2, 11].”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “To quantify undesired changes, we follow [6] to compute masked LPIPS [66].”
Unified Concept Editing in Diffusion Models (2023, background): “Closed-form solutions for editing knowlege in generative text models have been proposed in [25, 26], while [2, 30] have edited knowledge in text-to-image diffusion models by targeting either the cross-attention layers or the text-encoder layers.”
Transforming and Projecting Images into Class-conditional Generative Networks (2020, unknown): “, [55, 9, 34, 37, 55, 7, 30].”
BlobGAN: Spatially Disentangled Scene Representations (2022, methodology): “We experiment with k ∈ [5, 50] depending on the data considered.”
Erasing Concepts from Diffusion Models (2023, unknown): “In image synthesis, a generative adversarial network (GAN) can be edited using a handful of words [14], a few sketches [46], warping gestures [47], or copy-and-paste [2].”
Multi-Concept Customization of Text-to-Image Diffusion (2022, background): “A closely related line of work edits a generative model directly [6,18,72].”
Label-Free Concept Bottleneck Models (2023, unknown): “Model editing/debugging : Our approach is related to a range of works proposing ways to edit networks, such as (Bau et al., 2020; Wang et al., 2022) for generative vision models, (Bau et al., 2020) for classifiers, or (Meng et al., 2022; Mitchell et al., 2021) for language models.”
Locating and Editing Factual Knowledge in GPT (2022, methodology): “Bau et al. (2020) has observed that an optimal update of a linear associative memory will insert a new key-value pair ( k ∗ , v ∗ ) by solving a constrained least-squares problem with a simple closed form solution: by setting is the original matrix, and C = KK T is a constant that can be estimated…”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “Recent works by Bau et al. [172, 173] are interesting approaches towards achieving this goal.”
Editing Conditional Radiance Fields (2021, methodology): “Furthermore, 2D GAN-based editing methods fail to correctly modify the color of the object or maintain shape consistency across views, due to the lack of 3D representation.”