Two-Stream Convolutional Networks for Dynamic Texture Synthesis

By Matthew Tesfaldet, Marcus A. Brubaker, Konstantinos G. Derpanis. 2017

56 citations recorded by the source corpus.

We introduce a two-stream model for dynamic texture synthesis. Our model is based on pre-trained convolutional networks (ConvNets) that target two independent tasks: (i) object recognition, and (ii) optical flow prediction. Given an input dynamic texture, statistics of filter responses from the object recognition ConvNet encapsulate the per-frame appearance of the input texture, while statistics of filter responses from the optical flow ConvNet model its dynamics. To generate a novel texture, a randomly initialized input sequence is optimized to match the feature statistics from each stream of an example texture. Inspired by recent work on image style transfer and enabled by the two-stream model, we also apply the synthesis approach to combine the texture appearance from one texture with the dynamics of another to generate entirely novel dynamic textures. We show that our approach generates novel, high quality samples that match both the framewise appearance and temporal evolution of input texture. Finally, we quantitatively evaluate our texture synthesis approach with a thorough user study.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “[45] used two pretrained ConvNets trained for object recognition and optical flow prediction separately.”
Artsheets for Art Datasets (2021, background): “From mimicking artists’ styles to creating new artworks, ML technologies are employed to generate music [71, 72], visual arts [73, 74], writing [75], and several other art forms [76].”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “This work can be divided into two groups: (i) methods that exploit the statistics of dynamics textures [68, 12, 77, 79] and (ii) learning-based approaches [80, 83, 45, 21, 78].”
DeepLandscape: Adversarial Modeling of Landscape Video (2020, methodology): “Video generation/prediction works generally consider either videos with articulated objects/multiple moving objects [17, 18] or videos with weakly structured moving objects or dynamic textures such as clouds, grass, fire [14, 4].”
Artificial intelligence in the creative industries: a review (2020, methodology): “o been employed for rendering objects and scenes. This includes the synthesis of 3D views from motion capture or from monocular cameras (see Section 3.4.4), shading [49] and dynamic texture synthesis [50]. Creating realistic lighting in animation and visual eects has also beneted by combining tradit”