Learning-based Video Motion Magnification

By Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed A. Elgharib, Frédo Durand, William T. Freeman, Wojciech Matusik. 2018

193 citations recorded by the source corpus.

Video motion magnification techniques allow us to see small motions previously invisible to the naked eyes, such as those of vibrating airplane wings, or swaying buildings under the influence of the wind. Because the motion is small, the magnification results are prone to noise or excessive blurring. The state of the art relies on hand-designed filters to extract representations that may not be optimal. In this paper, we seek to learn the filters directly from examples using deep convolutional neural networks. To make training tractable, we carefully design a synthetic dataset that captures small motion well, and use two-frame input for training. We show that the learned filters achieve high-quality results on real videos, with less ringing artifacts and better noise characteristics than previous methods. While our model is not trained with temporal filters, we found that the temporal filters can be used with our extracted representations up to a moderate magnification, enabling a frequency-based motion selection. Finally, we analyze the learned filters and show that they behave similarly to the derivative filters used in previous works. Our code, trained model, and datasets will be available online.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, background): “Oh and Jaroensri et al. [53] build a synthetic dataset, which captures small motion for video motion magniﬁcation.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “If the training data is a rapid sequence of images frames in time [67] the objects of a frame can be assumed to move in a linear trajectory between consecutive frames.”
Preserving integrity in online social networks (2020, background): “In these cases, manipulated or full synthetic images lack some subtle cues like blood flow [85] or eye blinking [68], which can be detected by computer vision methods.”