M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

By Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, Hongxia Yang. 2021

49 citations recorded by the source corpus.

Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, M6-UFC, to unify any number of multi-modal controls. In M6-UFC, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, M6-UFC adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi-Modal CelebA-HQ verify that M6-UFC can synthesize high-fidelity images that comply with flexible multi-modal controls.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “Controlling elements additional to text have been suggested by [69], yet their use is con-ﬁned to restricted datasets such as fashion items or faces.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Recent advances in image quantization [53, 54] have demonstrated effectiveness in text-to-image synthesis [50, 18, 51, 19], and thus we utilize this strategy for the target-side image representations.”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “Some researches also show that multi-model methods can learn a robust generator from various translation tasks [64, 29, 19, 40].”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “Considering the synergy among tasks, some research efforts [53, 60, 26, 22, 38, 6, 42] aim to learn a uniﬁed model for diverse translation tasks via multi-task training.”
MAGVIT: Masked Generative Video Transformer (2022, methodology): “Concurrently, a few methods use non-autoregressive transformers for image synthesis [12, 42, 53, 76].”
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining (2021, background): “Furthermore, large-scale pretraining has recently demonstrated success in other fields, including pretraining on other languages or cross-lingual pretraining [44; 49; 47; 7], cross-modal pretraining [28; 32; 19; 10; 51] and code”
Masked Diffusion Transformer is a Strong Image Synthesizer (2023, background): “Sub-sequently, it also proved feasible for vision recognition [3] and generation [7,16,57] tasks.”
CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers (2022, methodology): “M6-UFC [38] ﬁrst introduces NAR methods into the VQ-VAE framework, and similar ideas are adopted by VQ-diffusion [11] and MaskGIT [1].”