NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

By Chenfei Wu, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, Nan Duan. 2021

369 citations recorded by the source corpus.

This paper presents a unified multimodal pre-trained model called N\"UWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate N\"UWA on 8 downstream tasks. Compared to several strong baselines, N\"UWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Compared with DALLE [50], CogView [51] and NÜWA [52], whose sampling sizes are 512 , 60 and 60 , respectively, OFA outperforms these SOTA methods on FID and IS with a much smaller sampling size 24 .”
MusicLM: Generating Music From Text (2023, methodology): “The towers map the two modalities to a shared embedding space of 128 dimensions using contrastive learning, with a setup similar to (Radford et al., 2021; Wu et al., 2022b).”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “…recipe can in principle be followed to express any input-output relationship, and breakthrough results have been achieved using Transformers (Vaswani et al., 2017) or related models to learn the input → output mapping (Vinyals et al., 2019; Brown et al., 2020; Jumper et al., 2021; Wu et al., 2021).”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, background): “Furthermore, NÜWA [77] unifies the training of generative models for images and videos using a frame-based VQGAN and adopts 3D nearby self-attention for better efficiency.”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, positive): “N ¨UWA [49] expands upon GODIVA by presenting a uni-ﬁed representation for various generation tasks through a multitask learning approach.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, neutral): “Method Zero-shot FVD ↓ CLIPSIM ↑ GODIVA [64] No - 0.2402 Nüwa [65] No - 0.2439 CogVideo (Chinese) [24] Yes - 0.2614 CogVideo (English) [24] Yes 1294 0.2631 MagicVideo [75] Yes 1290 - Make-A-Video [49] Yes - 0.3049 Video LDM [”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[99] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “In terms of memory consumption, our base, interpolation and VSR models require 13GB, 40GB and 42GB respectively.”
ModelScope Text-to-Video Technical Report (2023, methodology): “When evaluating on CLIPSIM metric, we followed previous works [51, 62] and use nearly 60k sentences from the whole test split as prompts to generate videos.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “In the subsequent super-resolution module, we modify the latent-based VDM and use our proposed expert translation to generate videos of 29 × 576 × 320.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “64 NUWA [177] arXiv-2021 image-text Trans T2I, T2V, V2V A 3D transformer framework can handle image, text, and video, simultaneously 809M URL”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Multiple classes of generative models have utilized Transformers [73] as back-bone, such as, Generative adversarial networks [42, 47, 85], autoregressive [10, 11, 20, 21, 27, 59, 74, 77, 78, 80, 81] and diffusion [2, 22, 41, 50, 53, 87] models.”