MaskGAN: Towards Diverse and Interactive Facial Image Manipulation

By Cheng-Han Lee, Ziwei Liu, Lingyun Wu, Ping Luo. 2019

1,287 citations recorded by the source corpus.

Facial image manipulation has achieved great progress in recent years. However, previous methods either operate on a predefined set of face attributes or leave users little freedom to interactively manipulate images. To overcome these drawbacks, we propose a novel framework termed MaskGAN, enabling diverse and interactive face manipulation. Our key insight is that semantic masks serve as a suitable intermediate representation for flexible face manipulation with fidelity preservation. MaskGAN has two main components: 1) Dense Mapping Network (DMN) and 2) Editing Behavior Simulated Training (EBST). Specifically, DMN learns style mapping between a free-form user modified mask and a target image, enabling diverse generation results. EBST models the user editing behavior on the source mask, making the overall framework more robust to various manipulated inputs. Specifically, it introduces dual-editing consistency as the auxiliary supervision signal. To facilitate extensive studies, we construct a large-scale high-resolution face dataset with fine-grained mask annotations named CelebAMask-HQ. MaskGAN is comprehensively evaluated on two challenging tasks: attribute transfer and style copy, demonstrating superior performance over other state-of-the-art methods. The code, models, and dataset are available at https://github.com/switchablenorms/CelebAMask-HQ.

Subfields: Computer Vision and Pattern Recognition, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, background): “One important class of CGANs use images as conditioning information, such as pix2pix (Isola et al. 2017), BicycleGAN (Zhu et al. 2017b), pix2pixHD (Wang et al. 2018), SPADE (Park et al. 2019), MaskGAN (Lee et al. 2019) and SEAN (Zhu et al. 2019).”
Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis (2020, methodology): “For this purpose, we conduct experiments on CelebA-HQ (Lee et al., 2020), which contains 30,000 aligned human face images regarding 40 attributes in total.”
Identity-Guided Face Generation with Multi-Modal Contour Conditions (2021, background): “Semantic labels and sketches are from [20] and [21], respectively.”
HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment (2020, methodology): “ntributions, as described below: •SPADEThe vanilla SPADE [44] network with semantic guidance being face parsing maps extracted from the original high-resolution images with a pretrained parsing model [28]. •16xFace replaces the semantic parsing map in SPADE with degraded faces containing 16-pixel mo”
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection (2021, background): “Beneﬁting from the great progress made in deep learning, the Variational AutoEncoders [21, 35] and Generative Adversarial Networks based [15] face manipulation technology [38, 23, 43] enables ordinary people without professional skills and equipment to generate high-quality forged faces.”
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP (2021, methodology): “In order to evaluate our method on face images that contain a bigger set of labels we test our method on CelebAMask-HQ dataset [9] segmentation face parsing dataset that contains 19 semantic classes.”
Taming Visually Guided Sound Generation (2021, background): “images [13, 17, 35, 41, 44, 46, 50, 52, 64, 66, 67], videos [6, 12, 25, 34, 38, 42, 59, 60, 60, 63], and audios [1, 9, 15, 22, 24, 43, 57, 58], or separate sounds [18, 19, 69, 70, 74].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “To further quantitatively evaluate the effectiveness of HeadNeRF, we randomly select 400 in-the-wild face images from the CeleAMask-HQ dataset [29] and the test dataset of FFHQ [21], respectively.”
BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models (2022, methodology): “In order to further verify the generalization of BBDM, we conducted inpainting, colorization experiments on VisualGENOME [17] and face-to-label on CelebAMaskHQ [18].”
Towards Photo-Realistic Virtual Try-On by Adaptively Generating↔Preserving Image Content (2020, background): “Beneﬁted from the powerful ability of GAN, it enjoys pervasive applications on tasks such as style transfer [52, 3], image inpainting [45, 15, 47, 48, 27], and image editing [20, 5, 24, 31].”
Low-Rank Subspaces in GANs (2021, methodology): “For segmentation models, [22] is used for face data, and PixelLib is used for LSUN.”
Intermediate Layer Optimization for Inverse Problems using Deep Generative Models (2021, methodology): “• Code for evaluating the performance of ILO and previous methods on Celeba-HQ (Liu et al., 2018; Lee et al., 2020).”
Generative Diffusion Prior for Unified Image Restoration and Enhancement (2023, methodology): “The CelebA-HQ [40] dataset is a high-quality subset of the Large-Scale CelebFaces Attributes (CelebA) dataset [49].”