Latent Space Oddity: on the Curvature of Deep Generative Models

By Georgios Arvanitidis, Lars Kai Hansen, Søren Hauberg. International Conference on Learning Representations, 2017

331 citations recorded by the source corpus.

Deep generative models provide a systematic way to learn nonlinear data distributions, through a set of latent variables and a nonlinear "generator" function that maps latent points into the input space. The nonlinearity of the generator imply that the latent space gives a distorted view of the input space. Under mild conditions, we show that this distortion can be characterized by a stochastic Riemannian metric, and demonstrate that distances and interpolants are significantly improved under this metric. This in turn improves probability distributions, sampling algorithms and clustering in the latent space. Our geometric analysis further reveals that current generators provide poor variance estimates and we propose a new generator architecture with vastly improved variance estimates. Results are demonstrated on convolutional and fully connected variational autoencoders, but the formalism easily generalize to other deep generative models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9f696b7156716c978b62a92714e7038a99f7a53c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CaSpeR: Latent Spectral Regularization for Continual Learning (2023, background): “Given the Riemannian nature of the latent space of DNNs [3], we naturally revert to spectral geometry to study its evolution.”
Manifold GPLVMs for discovering non-Euclidean latent structure in neural data (2020, background): “, 2014) and other latent variable models (Arvanitidis et al., 2017) after initially learning the models with a Euclidean latent.”
A survey of algorithmic recourse: definitions, formulations, solutions, and prospects (2020, background): “Finally, other works aim to minimize dissimilarity on a graph manifold [149], in terms of Euclidean distance in a learned feature space [87, 143], or using a Riemannian metric in a latent space [14, 15].”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, background): “Latent space of GANs is generally treated as Riemannian manifold [9, 2, 21].”
Is Generator Conditioning Causally Related to GAN Performance? (2018, background): “In a related vein, three similar papers (Arvanitidis et al., 2017; Chen et al., 2017; Shao et al., 2017) have explicitly studied variational autoencoders through the lens of geometry.”
Transferring Knowledge across Learning Processes (2018, background): “Related work studying models from a geometric perspective have explored how to interpolate in a generative model’s learned latent space (Tosi et al., 2014; Shao et al., 2017; Arvanitidis et al., 2018; Chen et al., 2018; Kumar et al., 2017).”
Distribution-Interpolation Trade off in Generative Models (2018, methodology): “Nonetheless, reducing the consequences of the distribution mismatch is still a popular research topic (Agustsson et al., 2017; Kilcher et al., 2017; Arvanitidis et al., 2017).”
Latent Space Non-Linear Statistics (2018, background): “[2] includes the z-variability of the variance σ(z) of VAEs resulting in the inclusion of the Jacobian of σ in the expected metric.”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, background): “Latent space of GANs is generally treated as Riemannian manifold [21], [22], [23].”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “…et al., 2016), which is theoretically appealing as it is tightly related to the log-likelihood of the original discrete data (Theis et al., 2016), optimizing the variance in a two-stage procedure (Arvanitidis et al., 2017), or training a post-hoc prior (Engel et al., 2017; Ghosh et al., 2019).”