ModelScope Text-to-Video Technical Report

By Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, Shiwei Zhang. arXiv.org, 2023

717 citations recorded by the source corpus.

This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/84f0a99d0f0015a6145c94468870d43ab1d166fd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “Regarding text-to-video (T2V) models, Make-A-Video [45] and Imagen Video [27] are cascaded models, while most other works, such as LVDM [24], Magic Video [63], Mod-elScope [50], and Align your Latents [10], are SD-based models.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Swayed by the success of diffusion models [24, 48, 100], a surge of diffusion-based video generation methods have emerged [6, 34, 46, 51, 108, 117, 121].”
Generative Multimodal Models are In-Context Learners (2023, methodology): “Similar to [46, 74], we adapt a 2D denoising U-Net to 3D style by inserting a 1D temporal convolution following each 2D spatial convolutional layer and extending the spatial attention to spatial-temporal attention.”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, neutral): “We evaluate 8 popular video generation model, including Open-Sora-Plan(T2V) (Lab & etc., 2024), Lavie (Wang et al., 2023c), ModelScope (Wang et al., 2023b), OpenSora (Zheng et al., 2024), Ani-mateDiff (Guo et al., 2023), Open-Sora-Plan(TI2V) (Lab & etc., 2024), DynamicCrafter (Xing et al., 2023),…”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “Recently, diffusion models [19, 38, 89, 91] have achieved significant progress in image synthesis [31, 46, 47, 78, 81, 84, 86], and enabled a line of works towards video generation [8, 13, 28, 33– 35, 39, 40, 50, 58, 72, 88, 98, 104, 118, 124, 132, 133].”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, neutral): “1, SD-XL, SD-XL Turbo), DeepFloyd-IF [13], Midjourney v6 [55], DALL-E 3 [2]; along with four text-to-video models: ModelScope [77], Floor33 [15], Pika v1 [59], Gen2 [18].”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “For text-to-video synthesis in particular, most works directly condition the model on a text prompt [9, 97] or make use of an additional text-to-image prior [23, 82].”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Current text-to-video generation methods can be divided into two categories from architecture design: UNet based methods [19, 20, 7, 21, 22, 23, 24], and DiT based methods [6, 8, 5].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “For evaluation, we generate videos using the provided prompts benchmark on five state-of-the-art methods of Mod-elScope [57], ZeroScope [5], Gen2 [16], Floor33 [1], and PikaLab [4], getting 2.5k videos in total.”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “We also incorporate the temporal convolution in Mod-elScopeT2V [48] to improve temporal consistency.”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, background): “…advances in computation, data scaling and architectural design, current visual generative models, especially diffusion-based models, have made remarkable strides in automating content creation, empowering designers to generate realistic images or videos from a textual prompt as input [22, 44, 49].”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, background): “VDM [15], VideoLDM [52] and Modelscope-T2V [44] achieve significant performance improvement by designing a UNet with temporal-awareness capability to directly generate complete video chunks.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “In terms of memory consumption, our base, interpolation and VSR models require 13GB, 40GB and 42GB respectively.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, positive): “To tackle this challenge, we introduced a space-time U-Net architecture design that directly generates full-frame-rate video clips, by incorporating both spatial, and temporal down-and up-sampling modules.”