Deep Feature Interpolation for Image Content Changes

By Paul Upchurch, Jacob V. Gardner, Geoff Pleiss, Kavita Bala, Robert Pless, Noah Snavely, Kilian Q. Weinberger. 2016

331 citations recorded by the source corpus.

We propose Deep Feature Interpolation (DFI), a new data-driven baseline for automatic high-resolution image transformation. As the name suggests, DFI relies only on simple linear interpolation of deep convolutional features from pre-trained convnets. We show that despite its simplicity, DFI can perform high-level semantic transformations like make older/younger, make bespectacled, add smile, among others, surprisingly well–sometimes even matching or outperforming the state-of-the-art. This is particularly unexpected as DFI requires no specialized network architecture or even any deep network to be trained for these tasks. DFI therefore can be used as a new baseline to evaluate more complex algorithms and provides a practical answer to the question of which image transformation tasks are still challenging after the advent of deep learning.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Grammar Variational Autoencoder (2017, background): “Generative machine learning models have been used recently to produce extraordinary results, from realistic musical improvisation (Jaques et al., 2016), to changing facial expressions in images (Radford et al., 2015; Upchurch et al., 2016), to creating realistic looking artwork (Gatys et al., 2015).”
Robustness May Be at Odds with Accuracy (2018, methodology): “…interpolation have thus far been restricted to generative models such as GANs (Goodfellow et al., 2014a) and VAEs (Kingma & Welling, 2013), involved manipulation of learned representations (Upchurch et al., 2016), and hand-designed methods (Suwajanakorn et al., 2015; Kemelmacher-Shlizerman, 2016).”
Fader Networks: Manipulating Images by Sliding Attributes (2017, methodology): “Finally, for the application of automatically modifying faces using attributes, the feature interpolation approach of [23] presents a means to generate alterations of images based on attributes using a pre-trained network on ImageNet.”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, background): “Some work has observed the vector arithmetic property [28, 33].”
An empirical study on evaluation metrics of generative adversarial networks (2018, background): “o a semantically meaningful feature space. For Kernel MMD, this corresponds to computing the usual inner product in the feature space ˚(X). Inspired by the Inception/Mode Score, and recent works from Upchurch et al. (2017); Larsen et al. (2015) which show that convolutional networks may linearize th”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, background): “This linear structure allows to explore the representations using vector arithmetics [49].”
On the "steerability" of generative adversarial networks (2019, methodology): “With Deep Feature Interpolation [24], one can remove the generative model entirely.”
Neural Style Transfer: A Review (2017, methodology): “delighted to find that related researches on NST also bring new inspirations for other areas [106], [107], [108] and accelerate the development of a wider vision community.”
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs (2020, background): “In particular, [23, 28] show that semantic manipulation is achievable by vector arithmetic on latent code; [20] show that style transfer can be achieved by mixing two latent represents.”
AttGAN: Facial Attribute Editing by Only Changing What You Want (2017, methodology): “There are two types of methods for facial attribute editing, the optimization based ones [18], [19] and the learning based ones [7]–[14], [16].”
Implicit Semantic Data Augmentation for Deep Networks (2019, background): “Deep networks are known to excel at forming high-level representations in the deep feature space [4, 5, 9, 27], where the semantic relations between samples can be captured by the relative positions of their features [10].”
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, background): “Also, Fader [41] Networks and 1https://github.com/alsgkals2/CoReD Deep Feature Interpolation [73] show promising results on manipulating facial features such as smile, age, and mustache.”
Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “This includes research ranging from uncovering emergent interpretable structures in intermediate representations [77, 46, 10], to the identification and modification of embedded knowledge [48, 42, 43], as well as steering model outputs [66, 7, 33, 24, 65].”
Class-Incremental Learning via Dual Augmentation (2021, background): “For instance, deep feature interpolation [40] leverages simple interpolations in the embedding space to achieve semantic augmentation.”
Towards Explanation of DNN-based Prediction with Guided Feature Inversion (2018, methodology): “[42] first linearly interpolated the higher layer of CNN and then inverted the modified feature to an image to perform high-level semantic transformations.”
Image Manipulation with Perceptual Discriminators (2018, methodology): “The results comply with the qualitative observations: artifacts, produced by DCGAN and DFI are being easily detected by the classifier resulting in a very low log-loss.”
Deep Network Interpolation for Continuous Imagery Effect Transition (2018, background): “[36] propose a linear interpolation of pre-trained deep convolutional features to achieve image content changes.”