Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation

By Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, Mike Zheng Shou. IEEE International Conference on Computer Vision, 2022

1,179 citations recorded by the source corpus.

To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator. Despite their promising results, such paradigm is computationally expensive. In this work, we propose a new T2V generation setting—One-Shot Video Tuning, where only one text-video pair is presented. Our model is built on state-of-the-art T2I diffusion models pre-trained on massive image data. We make two key observations: 1) T2I models can generate still images that represent verb terms; 2) extending T2I models to generate multiple images concurrently exhibits surprisingly good content consistency. To further learn continuous motion, we introduce Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy. At inference, we employ DDIM inversion to provide structure guidance for sampling. Extensive qualitative and numerical experiments demonstrate the remarkable ability of our method across various applications.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1367dcff4ccb927a5e95c452041288b3f0dd0eff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Based on these models, advanced image and video editing have been achieved through finetuning the model [41, 67, 5, 23, 61, 29] or controlling the inference process [30, 13, 34, 10, 35, 7, 31, 3].”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “Tune-A-Video [53] observed that by extending the self-attention module to operate on more then a single frame, it is possible to generate frames that share a common global appearance.”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “Inspired by the phenomenon that performing self-attention across batches can generate similar image contents, which is also observed in Tune-A-Video [37], we adapt the self-attention mechanism in the T2I model to query contents from source images with delicate designs.”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, methodology): “Unlike AnimateDiff [8] (which refers to the motion control LoRA model provided by AnimateDiff), Tune-a-video [26], LAMP [27], and MotionDirector [30] that implement motion control by extracting motion from one or a series of template videos and require different models for different template…”
Video-P2P: Video Editing with Cross-Attention Control (2023, positive): “Similar to TAV [47], we assume that the object of interest is present in the ﬁrst frame.”
SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, positive): “It has been demonstrated that T2I diffusion priors are powerful in image editing [42, 72], video generation [50, 62], 3D content generation [37, 60, 60], etc .”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, background): “Text-based generation has been extended to other modalities including audio [35, 17, 28, 70], video [75, 91, 85, 25], and 4D fields [76].”
Pix2Video: Video Editing using Image Diffusion (2023, positive): “Jamriska et al. [20] no no no Text2Live [3] 7-8 hours 30 min yes Tune-a-Video [58] 75sec 10 min no ours 125sec no no “a grizzly bear walking around the lake, foggy day” “a group of rainbow fish swimming in an aquarium” “a kite-surfer in the magical starry ocean with aurora borealis in the…”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “In this study, we present StableSR , an approach that preserves pre-trained diffusion priors without making explicit assumptions about the degradations.”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “…editing methods and tune-a-video, our method can edit the style of the real-world video and images using the trained text-to-image model [42] and shows better object replacing performance than the one-shot ﬁnetuned video diffusion model [52] with simple DDIM inversion [46] in real videos (Fig.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Recent works (e.g., VoP [88], Make-A-Video [28], and Tune-A-Video [89]) have shown how prompt engineering leverages model’s nat-ural language understanding ability to decode complex instructions and render them into cohesive, lively, and high-quality video narratives.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “Numerous studies leverage PEFT techniques to adapt a pre-trained diffusion model for downstream tasks, including accelerating sampling speed [125], [11], text-to-video adaptation [186], [191], text-to-3D adaptation [209], etc.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, neutral): “…into applying diffusion models for both conditional and unconditional video synthesis (Voleti et al., 2022; Harvey et al., 2022; Zhou et al., 2022; Wu et al., 2022b; Blattmann et al., 2023b; Khachatryan et al., 2023; H¨oppe et al., 2022; Voleti et al., 2022; Yang et al., 2022; Nikankin et al.,…”
ControlVideo: Training-free Controllable Text-to-Video Generation (2023, methodology): “Rather than synthesizing all frames independently, [15,40] enhance appearance coherence by replacing original self-attention with the sparser cross-frame attention.”
Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “For systematic benchmarking, recent work [3, 4, 39, 64, 68, 80] adopts metrics such as CLIPScore [21, 61], which measures the (cosine) similarity of the embedded image and text prompt.”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, background): “For instance, Tune-a-Video [31] solves the one-shot video generation task via slight architecture modiﬁcations and sub-network tuning.”
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, positive): “Specifically, we inflate the original 2D UNet to 3D temporal UNet by inserting temporal attention layers [10, 40, 53].”