GANalyze: Toward Visual Definitions of Cognitive Image Properties

By Lore Goetschalckx, Alex Andonian, Aude Oliva, Phillip Isola. 2019

350 citations recorded by the source corpus.

We introduce a framework that uses Generative Adversarial Networks (GANs) to study cognitive properties like memorability. These attributes are of interest because we do not have a concrete visual definition of what they entail. What does it look like for a dog to be more memorable? GANs allow us to generate a manifold of natural-looking images with fine-grained differences in their visual attributes. By navigating this manifold in directions that increase memorability, we can visualize what it looks like for a particular generated image to become more memorable. The resulting ``visual definitions" surface image properties (like ``object size") that may underlie memorability. Through behavioral experiments, we verify that our method indeed discovers image manipulations that causally affect human memory performance. We further demonstrate that the same framework can be used to analyze image aesthetics and emotional valence. ganalyze.csail.mit.edu.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

In Safety, Robustness & Interpretability

Citations in the corpus

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “Then,w can be manipulated to achieve semantically meaningful edits [Goetschalckx et al. 2019; Shen et al. 2020].”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, background): “[14] improves the memorability of the output image.”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “While many previous approaches to solving image-to-image translation tasks involve dedicated architectures spe-ciﬁc for solving a single problem, we follow the spirit of pix2pix [16] and deﬁne a generic framework able to solve a wide range of tasks, all using the same architecture.”
Paint by Word (2021, background): “Early work on GANs observed the presence of some latent vector directions corresponding to semantic concepts [32], and further explorations have developed methods to identify interesting vector directions that steer latent space by using reconstruction losses [19, 37] or by learning from classificat”
GANSpace: Discovering Interpretable GAN Controls (2020, background): “Current attempts to add user control over the output focus on supervised learning of latent directions [9, 7, 25, 19, 16], GAN training with labeled images [12, 22, 21].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “However, current methods require either a pretrained classifier [10, 28, 29, 34], a large set of paired examples [15], or manual examination of many candidate control directions [12], which limits the versatility of these approaches.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, methodology): “[10] uses a similar approach to ﬁnd attribute vectors associated with cognitive properties such as memorability, aesthetics and emotional valence.”
On the "steerability" of generative adversarial networks (2019, background): “Goetschalckx et al. (2019) applies a linear walk to achieve transformations corresponding to cognitive properties of an image such as memorability, aesthetics, and emotional valence.”
Closed-Form Factorization of Latent Semantics in GANs (2020, methodology): “which is commonly used in the existing approaches [7, 24, 27, 26, 10].”
Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs (2021, background): “Some studies have investigated the supervised learning of latent directions while assuming access to the semantic attributes of images (Goetschalckx et al., 2019; Jahanian et al., 2019; Shen et al., 2020; Yang et al., 2021; Abdal et al., 2021).”
Only a matter of style: age transformation using a style-based regression model (2021, methodology): “Other works [Abdal et al. 2020b; Härkönen et al. 2020; Liu et al. 2020; Shen et al. 2020; Wu et al. 2020] have approached the age transformation task by exploring the semantics of the latent space of a well-trained GAN, such as StyleGAN [Karras et al.”
Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs (2020, background): “For the same class of models, [19] finds directions in their input space which represent semantic concepts corresponding to certain cognitive properties.”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “Some using fullsupervision in the form of semantic labels [10, 11, 34], others [15, 30, 35] find meaningful directions in a selfsupervised fashion, and finally recent works present unsupervised methods to achieve the same goal [14, 39, 40], requiring no manual annotations.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “They either modify the training objective [9, 40, 71] or network architecture [39], or investigate latent spaces of well-engineered and pre-trained generative models [1, 16, 23, 27, 34, 78, 96].”
GAN Steerability without optimization (2020, background): “, (Shen et al., 2020; Radford et al., 2015; Karras et al., 2018; 2019b; Denton et al., 2019; Xiao et al., 2018; Goetschalckx et al., 2019)).”
Controlling generative models with continuous factors of variations (2020, background): “The closest works were released on ArXiv very recently (Goetschalckx et al., 2019; Jahanian et al., 2019) indicating that finding interpretable directions in the latent space of generative models to control their output is of high interest for the community.”