Structure and Content-Guided Video Synthesis with Diffusion Models

By Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, Anastasis Germanidis. IEEE International Conference on Computer Vision, 2023

760 citations recorded by the source corpus.

Text-guided generative diffusion models unlock powerful image creation and editing tools. Recent approaches that edit the content of footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames.In this work, we present a structure and content-guided video diffusion model that edits videos based on descriptions of the desired output. Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects. As a solution, we show that training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity. A novel guidance method, enabled by joint video and image training, exposes explicit control over temporal consistency. Our experiments demonstrate a wide variety of successes; fine-grained control over output characteristics, customization based on a few reference images, and a strong user preference towards results by our model.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/07be0ec1f45e21a1032616535d0290ee6bfe0f6b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Genie: Generative Interactive Environments (2024, neutral): “Early signs indicate video generation will be yet another frontier, with recent results suggesting that such models may also benefit from scale (Blattmann et al., 2023a; Esser et al., 2023; Ho et al., 2022a; Hong et al., 2023).”
Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators (2023, methodology): “Gen-1 [5] extends SD and proposes a structure and content-guided video editing method based on visual or textual descriptions of desired outputs.”
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, methodology): “4) Frame Consistency: Following [6], we report frame consistency measured via CLIP cosine similarity of consecutive frames, and the results are shown in Tab.”
Generative artificial intelligence (2023, positive): “New video clips are generated by describing the content of the desired video footage (text-to-video) or applying the style and composition via text or image prompt to a source video (video-to-video) (Esser et al., 2023).”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, methodology): “Typically, Gen-1 [8] utilizes depth as an additional condition to decouple structure and content and achieves excellent results in video-to-video translation tasks.”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, positive): “Tune-A-Video Text-to-Video An intuitive solution is to train a T2V model on large-scale video datasets, but it is computationally expensive [43, 16, 7].”
VideoComposer: Compositional Video Synthesis with Motion Controllability (2023, methodology): “A subsequent work, Gen-1 [15], integrates depth maps alongside texts using cross-attention mechanisms to provide structural guidance.”
FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, methodology): “Specially, we show the ‘Tem-Con’ [9] to measure the temporal consistency in frames by computing the cosine similarity between all pairs of consecutive frames.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “11, we present visual comparisons of Show-1 with four other methods such as Gen-1 (Esser et al., 2023) and Tune-A-Video (Wu et al., 2022b).”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “Some approaches seek to replace the guidance signal from text to image, focusing on the I2V setting (Chen et al., 2023b;d; Esser et al., 2023).”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “…generation, including joint training on the text-image data from scratch [17, 14, 56, 60], first training a text-to-image model and then finetuning partially [18, 4, 61, 29] or entirely [46, 8] on the video dataset, and using CLIP image features as the conditional information [46, 69].”
ModelScope Text-to-Video Technical Report (2023, methodology): “Conversely, during inference, random noise Z pr 0 is sampled and utilized for the denoising procedure. decoupling, Gen-1 [8] defines the content latents as CLIP embeddings and the structure latents as the monocular depth estimates, attaining superior decoupled controllability.”
TokenFlow: Consistent Diffusion Features for Consistent Video Editing (2023, methodology): “We compare our method against Tune-A-Video [53], PnP-Diffusion [51] applied per frame, Gen-1 [10], and Text2Video-Zero[20].”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “For text-to-video synthesis in particular, most works directly condition the model on a text prompt [9, 97] or make use of an additional text-to-image prior [23, 82].”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, neutral): “For instance, AnimateDiff [8], Gen-2 [7], and PikaLab [2] mainly execute or trigger camera motion control using independent LoRA [11] models or extra camera parameters (such as “-camera zoom in” in PikaLab [2]).”
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, methodology): “Recent general text-to-video generation approaches [7,12,33] pro-pose incorporating temporal modeling into the original T2I models and tuning the models on the video datasets.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Similar to adding controls in T2I [48, 61, 88, 93], introducing control signals in T2V, such as structure [17, 83], pose [44, 94], and Canny edge [39], has been increasingly receiving much attention.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Inspired by latent video diffusion models [2, 13, 48, 73], we introduce multiview and temporal modeling for jointly generating multiple views and frames.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “They can operate either at the pixel level [38, 17, 66, 67, 68, 69, 16] or in the latent space of a pre-trained image tokenizer [70, 18, 71].”