Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation

By David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, L. Ran, Yuchao Gu, Difei Gao, Mike Zheng Shou. International Journal of Computer Vision, 2023

353 citations recorded by the source corpus.

Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution, which can also remove potential artifacts and corruptions from low-resolution videos. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15 G vs. 72 G). Furthermore, our Show-1 model can be readily adapted for motion customization and video stylization applications through simple temporal attention layer finetuning. Our model achieves state-of-the-art performance on standard video generation benchmarks. Code of Show-1 is publicly available and more videos can be found here.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/a5b7fc1bff0910ff31975ec0a15ed30c41f0a968

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “We will open-source our VBench and encourage more T2V models to participate in the evaluation, including but not limited to [1–4, 8, 124], so that we can provide more informed insights into the current state of T2V, and provide more annotated data on T2V generation results generated by different…”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “We compare our approach with several state-of-the-art T2V models, including popular commercial models such as Gen-2 [5] and Pika Labs [9], as well as open-source models like Show-1 [60], VideoCrafter1 [15], and AnimateD-iff [22].”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, neutral): “…top-performing text-to-video models as baselines, including T2V-Turbo (Li et al., 2024), AnimateDiff (Guo et al., 2023), VideoCrafter2 (Chen et al., 2024a), OpenSora (Zheng et al., 2024b), Show-1 (Zhang et al., 2023a), Gen-2 (runway, 2023), Pika (pik, 2023), and LaVie-2 (Wang et al., 2023b).”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “Due to the success of diffusion models in the field of image generation [72, 34], the exploration in the domain of video generation [28, 40, 75, 84, 91, 96, 58, 12, 90, 57] is also becoming popular.”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “Similar methods are also been proposed by SHOW-1 [65] and LAVIE [59] T2V generation also raises the enthusiasm of commerce or non-commerce companies.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “Other works (Guo et al., 2023; Zhang et al., 2024) focus on constructing the personalized text-to-video model to partially train temporal modules which can be adapted to various different personalized text-to-image models for personalized video generation.”