Guided Image Generation with Conditional Invertible Neural Networks

By Lynton Ardizzone, Carsten T. Lüth, Jakob Kruse, Carsten Rother, Ullrich Köthe. arXiv.org, 2019

351 citations recorded by the source corpus.

In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). The cINN combines the purely generative INN model with an unconstrained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of the cINN are jointly optimized with a stable, maximum likelihood-based training procedure. By construction, the cINN does not experience mode collapse and generates diverse samples, in contrast to e.g. cGANs. At the same time our model produces sharp images since no reconstruction loss is required, in contrast to e.g. VAEs. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bi-directional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3c3698ce314a77271b566713edb2911f921299e3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, background): “Because of its lossless information-preserving property, INNs have been effectively integrated into image processing ﬁelds such as image coloring [3], image hiding [23], image rescaling [68] and image/video super-resolution [94,95].”
Palette: Image-to-Image Diffusion Models (2021, background): “Colorization is awell-studied task [Ardizzone et al. 2019; Guadarrama et al. 2017; Kumar et al. 2021; Royer et al. 2017], requiring a degree of scene understanding, which makes it a natural choice for self-supervised learning [Larsson et al. 2016].”
SRFlow: Learning the Super-Resolution Space with Normalizing Flow (2020, methodology): “tly minimizing the negative log-likelihood using standard SGD-based techniques. Recent works have investigated conditional ow models for point cloud generation [37,51] as well as class [25] and image [3,49] conditional generation of images. The latter works [3,49] adapt the widely successful Glow ar”
CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows (2021, methodology): “We extend unconditional flow framework to CFLOW by concatenating the intermediate vectors inside decoder coupling layers with the conditional vectors ci as in [2].”
Disentanglement through Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, background): “A similar network design to ours is Ardizzone et al. (2019). This is a conditioned INN based on RealNVP, however the conditioning information is applied as a parameter of the network.”
Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, background): “A similar network design to ours is Ardizzone et al. (2019).”
Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (2021, methodology): “Different from above flowsteps, we use conditional affine coupling layer [3, 45] rather than ordinary affine coupling layer to obtain a conditional flow.”
Improving Diffusion Models for Inverse Problems using Manifold Constraints (2022, methodology): “We use pre-trained score functions from [25] 0.948 0.154 0.957 0.182 cINN [2] 0.952 0.166 0.952 0.180 pix2pix [21] 0.935 0.184 0.947 0.174 Inpainting Score-SDE [41], REPAINT [32], DDRM [25] were chosen as baseline diffusion models to compare against the proposed method.”
Colorization Transformer (2021, methodology): “We obtain the FID scores from (Ardizzone et al., 2019) and the human evaluation results from (Guadarrama et al., 2017).”
BayesFlow: Learning Complex Stochastic Models With Invertible Neural Networks (2020, background): “Invertible neural networks are based on the recent theory and applications of normalizing flows [3, 29, 20, 13, 27].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Normalizing flows are a natural fit for likelihood-free inference, especially flows conditioned on side information (e.g. Winkler et al., 2019; Ardizzone et al., 2019).”
Stochastic Image-to-Video Synthesis using cINNs (2021, methodology): “In every affine coupling layer, we additionally insert the conditioning information following previous work [2, 66].”
Low-Light Image Enhancement with Normalizing Flow (2021, methodology): “Recently, conditional afﬁne coupling layer (Ardizzone et al. 2019; Winkler et al. 2019; Lugmayr et al. 2020) is used to build a stronger connection with the conditional feature and improve the ef-ﬁciency of memory and computational resource.”