Artistic Style Transfer for Videos and Spherical Images

By Manuel Ruder, Alexey Dosovitskiy, Thomas Brox. 2017

160 citations recorded by the source corpus.

Manually re-drawing an image in a certain artistic style takes a professional artist a long time. Doing this for a video sequence single-handedly is beyond imagination. We present two computational approaches that transfer the style from one image (for example, a painting) to a whole video sequence. In our first approach, we adapt to videos the original image style transfer technique by Gatys et al. based on energy minimization. We introduce new ways of initialization and new loss functions to generate consistent and stable stylized video sequences even in cases with large motion and strong occlusion. Our second approach formulates video stylization as a learning problem. We propose a deep network architecture and training procedures that allow us to stylize arbitrary-length videos in a consistent and stable way, and nearly in real time. We show that the proposed methods clearly outperform simpler baselines both qualitatively and quantitatively. Finally, we propose a way to adapt these approaches also to 360∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document} images and videos as they emerge with recent virtual reality hardware.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “To constrain the consistency between the generated subsequent frames, a shift loss is added to the optimization objective, which is inspired by the artistic style transfer for videos literature (Ruder et al., 2017).”
Improving Data‐Driven Global Weather Prediction Using Deep Convolutional Neural Networks on a Cubed Sphere (2020, background): “Applying standard two-dimensional convolution operations within CNNs on faces of a cubed sphere is not unprecedented, as there are a number of examples of “cube map” convolutions applied to 360◦ images and videos within the computer science literature [36, 27, 23].”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “, [30] for optimization based video style transfer, [31, 1, 12, 13, 8] for per-style-per-model methods, [37, 38] for arbitrary-style-per-model methods, and [36, 4, 26] for image-to-image translation frameworks.”
SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images (2018, background): “One way of mitigating the problem of learning spherical representations are cube map projections as considered in [19, 22].”
VR-Goggles for Robots: Real-to-Sim Domain Adaptation for Visual Control (2018, background): “2 validate the discussion from [23], since the stylizations for s and s[x→4] from FF are almost identical (K = 4 for the trained network), but differ substantially otherwise.”