COCO-GAN: Generation by Parts via Conditional Coordinating

By Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, Hwann-Tzong Chen. 2019

149 citations recorded by the source corpus.

Humans can only interact with part of the surrounding environment due to biological restrictions. Therefore, we learn to reason the spatial relationships across a series of observations to piece together the surrounding environment. Inspired by such behavior and the fact that machines also have computational constraints, we propose COnditional COordinate GAN (COCO-GAN) of which the generator generates images by parts based on their spatial coordinates as the condition. On the other hand, the discriminator learns to justify realism across multiple assembled patches by global coherence, local appearance, and edge-crossing continuity. Despite the full images are never manipulated during training, we show that COCO-GAN can produce state-of-the-art-quality full images during inference. We further demonstrate a variety of novel applications enabled by our coordinate-aware framework. First, we perform extrapolation to the learned coordinate manifold and generate off-the-boundary patches. Combining with the originally generated full image, COCO-GAN can produce images that are larger than training samples, which we called "beyond-boundary generation". We then showcase panorama generation within a cylindrical coordinate system that inherently preserves horizontally cyclic topology. On the computation side, COCO-GAN has a built-in divide-and-conquer paradigm that reduces memory requisition during training and inference, provides high-parallelism, and can generate parts of images on-demand.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Taming Transformers for High-Resolution Image Synthesis (2020, background): “unconditional image synthesis on aligned data, we can simply condition on image coordinates, similar to [38].”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “Plenty of efforts have been made to improve the training and performance of GANs, from reformulation of the objective function [31, 1, 26, 37], integration of different regularization techniques [51, 34, 40, 48] and architectural changes [38, 19, 13, 27].”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, neutral): “42 COCO-GAN (Lin et al., 2019) 4.”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, positive): “Grid representation have been widely used in vision tasks, including self-supervised learning (Oord et al., 2018; Henaff et al., 2019; Trinh et al., 2019; Gidaris et al., 2020; Noroozi and Favaro, 2016) and image generation (van den Oord et al., 2017; Lin et al., 2019).”
InOut: Diverse Image Outpainting via GAN Inversion (2021, methodology): “1, we first introduce a generator based on the StyleGAN [14, 15] and COCO-GAN [18].”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, unknown): “…(Dai & Wipf, 2018) 44.4 WAE (Tolstikhin et al., 2018) 35 Perceptial AE(Zhang et al., 2020) 13.8 Latent EBM (Pang et al., 2020) 37.87 COCO-GAN (Lin et al., 2019) 4.0 QA-GAN (Parimala & Channappayya, 2019) 6.42 NVAE-Recon (Vahdat & Kautz, 2020) 1.03 (Vahdat & Kautz, 2020) 51.71 RAE (Ghosh et…”
Musika! Fast Infinite Waveform Music Generation (2022, unknown): “…, 256 , 512] and we always choose fft _ size = 4 · hop _ size , while the discriminator takes as input log-magnitude spectrograms calculated with [29, 30], two consecutive reconstructed spectro-grams are concatenated along the temporal dimension and fed to the discriminator, such that…”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, methodology): “As shown in Table 1, TransGAN largely outperforms Progressive-GAN [16] and COCOGAN [74], and is slightly better than the strongest competitor StyleGAN-v2 [40], by reaching a FID score of 5.”
Score Matching Model for Unbounded Data Score (2021, unknown): “…t ) and 1 σ (advantageous over log σ on small t ) as 1.90 DDPM [5] 4.90 Likelihood-free Models StyleGAN [1] 2.65 INR-GAN-bil [50] 4.95 COCO-GAN [51] 6.95 3.31 Anycost GAN [53] 3.35 StyleGAN2 ADA+bCR [28] 3.62 CIPS [54] 4.38 U-Net GAN [55] 7.48 BigGAN [55] 11.48 datasets, and 0.15 on 1024 ×…”
Dual Contrastive Loss and Attention for GANs (2021, background): “This often leads to the generated samples with discontinued semantic structures [51, 98] or the generated distribution with mode collapse [72, 96].”
StyleSwin: Transformer-based GAN for High-resolution Image Generation (2021, background): “, mouth, highly depends on its spatial coordinate [1, 40].”
InfinityGAN: Towards Infinite-Resolution Image Synthesis (2021, methodology): “A different approach, COCO-GAN [17], learns a coordinate-conditioned patch distribution for image synthesis.”