Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

By Chuan Li, Michael Wand. 2016

811 citations recorded by the source corpus.

This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN feature pyramid, controlling the image layout at an abstract level. We apply the method to both photographic and non-photo-realistic (artwork) synthesis tasks. The MRF regularizer prevents over-excitation artifacts and reduces implausible feature mixtures common to previous dCNN inversion approaches, permitting synthesizing photographic content with increased visual plausibility. Unlike standard MRF-based texture synthesis, the combined system can both match and adapt local features with considerable variability, yielding results far out of reach of classic generative MRF methods.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (2016, methodology): “Recently, Li and Wand [31] also investigated the effect of comparing and blending patches in pixel or VGG feature space.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “Other effective loss functions have been proposed, such as MRF loss [30], adversarial loss [31], histogram loss [54], CORAL loss [41], MMD loss [33], and distance between channel-wise mean and variance [33].”
Amortised MAP Inference for Image Super-resolution (2016, background): “Bruna et al. (2016) and Li & Wand (2016) measure the Euclidean distance in the nonlinear feature space of a deep NN pre-trained to perform object classiﬁcation.”
DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, methodology): “Several prior works have used gradient ascent on a DNN’s intermediate layers’ outputs to maximize those layers’ activation for transferring the artistic styles of one image to the other [17, 29, 39].”
Generative Image Inpainting with Contextual Attention (2018, background): “Several works [35, 38] follow ideas from image stylization [5, 24] to formulate the inpainting as an optimization problem.”
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis (2016, methodology): “More recently, deep neural network is introduced for texture synthesis and image stylization [15, 16, 28, 3, 39, 22].”
Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis (2017, background): “Recently, modiﬁed GAN architectures, conditional GAN [21] in particular, have been successfully applied to vision tasks like image inpainting [24], super-resolution [19], style transfer [20], face attribute manipulation [29] and even data augmentation for boosting classiﬁcation models [30,43].”
Fast Patch-based Style Transfer of Arbitrary Style (2016, methodology): “An alternative approach uses patch-based similarity matching between the content and style images [9, 10, 21].”
Contextual-Based Image Inpainting: Infer, Match, and Translate (2017, methodology): “Li and Wand [10] optimize the output image such that each of the neural patch matches with a similar neural patch in the style image.”
Universal Style Transfer via Feature Transforms (2017, methodology): “Numerous methods have since been developed to address different aspects including speed [27, 19, 16], quality [28, 18, 32, 31], user control [10], diversity [29, 20], semantics understanding [7, 1] and photorealism [23].”
Deep Photo Style Transfer (2017, methodology): “[11], CNNMRF [10], and Neural Style [5].”
Image Inpainting via Generative Multi-column Convolutional Neural Networks (2018, methodology): “Following common practice [5, 14], we use conv4_2 to describe image semantic structures.”
Neural Style Transfer via Meta Networks (2018, background): “There are also other representations of the style, such as histogram loss [30], MRF loss [20] and CORAL loss [29].”