Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions

By Ricard Durall, Margret Keuper, Janis Keuper. Computer Vision and Pattern Recognition, 2020

541 citations recorded by the source corpus.

Generative convolutional deep neural networks, e.g. popular GAN architectures, are relying on convolution based up-sampling methods to produce non-scalar outputs like images or video sequences. In this paper, we show that common up-sampling methods, i.e. known as up-convolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly. This effect is independent of the underlying architecture and we show that it can be used to easily detect generated data like deepfakes with up to 100% accuracy on public benchmarks. To overcome this drawback of current generative models, we propose to add a novel spectral regularization term to the training optimization objective. We show that this approach not only allows to train spectral consistent GANs that are avoiding high frequency errors. Also, we show that a correct approximation of the frequency spectrum has positive effects on the training stability and output quality of generative networks.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/4df3b534beeba05ba930814162ea6e19948c5fcd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image (2023, methodology): “Durall et al. [6] show that the up-sampling methods, such as the up-convolution or transposed convolution lead to the inability of GANs to correctly approximate the spectral distributions of training data, which explains why the artifacts occur in GANs.”
Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain (2021, background): “Though [11] also tried to detect these artifacts with the amplitude spectrum, the performance is limited due to the information loss.”
Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Domain Learning (2024, background): “The research by (Frank et al. 2020; Durall et al. 2020), which highlights the effectiveness of frequency artifacts in the domain of deepfake detection, has significantly influenced subsequent studies.”
Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “, perceptual loss [11] and spectral regularization [4], we follow all the details in their papers and released code.”
Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection (2023, background): “Some studies [16, 30, 45] focus on human face images, while others [11,18,19,42] handle various categories of images.”
Media Forensics and DeepFakes: An Overview (2020, background): “fail to reproduce spectral distributions of natural training data correctly [174].”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “Durall et al. (2020) showed that up-scaling operations commonly used in GANs ( e.g. up-convolutions) alter the spectral properties of the images causing high frequency distortions in the output.”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “Similarly, the second method is concerned with improving the quality of GAN-generated samples by enhancing spectral distributions [283].”
Dual Contrastive Loss and Attention for GANs (2021, background): “Yet, behind the seemingly saturated performance of the state-of-the-art StyleGAN2 [43], there still persists open issues of GANs that make generated images surprisingly obvious to spot [95, 81, 21, 28].”
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis (2021, unknown): “TransUNet uses only a transformer in its bottleneck while propagating shallow convolutional features via encoderdecoder skip connections, and its decoder increases spatial resolution via bilinear upsampling that might be ineffective in suppressing high-frequency artifacts [103].”
Detection of AI-Generated Synthetic Faces (2022, methodology): “It is worth noting that both approaches propose also countermeasures to limit the appearance of such spectral artifacts by means of a simple post-processing [17] or a spectral loss to be used during GAN training [16].”