Transframer: Arbitrary Frame Prediction with Generative Models

By Charlie Nash, João Carreira, Jacob Walker, Iain Barr, Andrew Jaegle, Mateusz Malinowski, Peter W. Battaglia. 2022

47 citations recorded by the source corpus.

We present a general-purpose framework for image modelling and vision tasks based on probabilistic frame prediction. Our approach unifies a broad range of tasks, from image segmentation, to novel view synthesis and video interpolation. We pair this framework with an architecture we term Transframer, which uses U-Net and Transformer components to condition on annotated context frames, and outputs sequences of sparse, compressed image features. Transframer is the state-of-the-art on a variety of video generation benchmarks, is competitive with the strongest models on few-shot view synthesis, and can generate coherent 30 second videos from a single image without any explicit geometric information. A single generalist Transframer simultaneously produces promising results on 8 tasks, including semantic segmentation, image classification and optical flow prediction with no task-specific architectural components, demonstrating that multi-task computer vision can be tackled using probabilistic image models. Our approach can in principle be applied to a wide range of applications that require learning the conditional structure of annotated image-formatted data.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Video Diffusion Models (2022, neutral): “Method FVD ↓ DVD-GAN [14] 109.8 VideoGPT [62] 103.3 TrIVD-GAN-FP [33] 103.3 Transframer [35] 100 CCVS [31] 99 VideoTransformer [59] 94 FitVid [4] 93.6 NUWA [61] 86.9 Video Diffusion (ours) ancestral sampler, 512 steps 68.19 Langevin sampler, 256 steps 66.92 Table 3: Video prediction on Kinetics-600.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, background): “…: A large blob of exploding splashing rainbow paint, with an apple emerging, 8k Parti [74] and [19]) and text-to-video ( e.g ., [36]), language models have not reached a level of quality on par with video diffusion models in tasks like text-to-video generation as shown in previous studies [42, 65].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “In parallel to our work, [22] proposes a gradient conditioning method for sampling longer videos with a diffusion model, [47] and [24] explore a hierarchical model with frame-level VQVAE embeddings or RGB frames.”
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (2022, methodology): “In [33] the authors propose a Transframer model, which uses a language model for modeling image outputs represented as sparse discrete cosine transform codes.”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “…networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] ﬂow based models [24], and ﬁnally adversarial based methods [48, 39,…”