Learning to Extract Motion from Videos in Convolutional Neural Networks

By Damien Teney, Martial Hebert. 2016

72 citations recorded by the source corpus.

This paper shows how to extract dense optical flow from videos with a convolutional neural network (CNN). The proposed model constitutes a potential building block for deeper architectures to allow using motion without resorting to an external algorithm, e.g. for recognition in videos. We derive our network architecture from signal processing principles to provide desired invariances to image contrast, phase and texture. We constrain weights within the network to enforce strict rotation invariance and substantially reduce the number of parameters to learn. We demonstrate end-to-end training on only 8 sequences of the Middlebury dataset, orders of magnitude less than competing CNN-based motion estimation methods, and obtain comparable performance to classical methods on the Middlebury benchmark. Importantly, our method outputs a distributed representation of motion that allows representing multiple, transparent motions, and dynamic textures. Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks (2016, neutral): “Following FlowNet, several papers have studied optical flow estimation with CNNs: featuring a 3D CNN [30], unsupervised learning [1, 33], carefully designed rotationally invariant architectures [28], or a pyramidal approach based on the coarse-to-fine idea of variational methods [20].”
Exploiting Cyclic Symmetry in Convolutional Neural Networks (2016, methodology): “A similar approach is investigated by Teney & Hebert (2016), where ﬁlters of individual convolutional layers are constrained to be rotations of each other.”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Our research is inspired by the success of applying deep learning to optical ﬂow estimation [2, 12, 16, 19, 50, 51, 52], artistic style transfer [17, 22, 28], and image enhancement [6, 9, 10, 46, 47, 55, 57, 62, 65].”
Detection and Characterization of the Fetal Heartbeat in Free-hand Ultrasound Sweeps with Weakly-supervised Two-streams Convolutional Networks (2017, methodology): “Contributions: In this paper, we propose an end-to-end trainable two-stream Convolutional Network (ConvNet), inspired by [2,3], for fetal heart recognition and characterization in clinical free-hand fetal US videos.”
Learning rotation invariant convolutional filters for texture classification (2016, background): “The authors of [18] propose to tie the weights within several layers of a deep CNN by splitting the filters into orientation groups.”
Deep Rotation Equivariant Network (2017, methodology): “Basically, the idea of most of the related works [9], [14], [15] is to stack rotated copies of images or features to obtain rotation equivariance.”
Video Frame Interpolation via Adaptive Convolution (2017, methodology): “Our work is inspired by the success of deep learning in solving not only difficult visual understanding problems [16, 20, 26, 28, 39, 40, 42, 45, 54, 60, 64] but also other computer vision problems like optical flow estimation [9, 14, 19, 48, 49, 52], style transfer [11, 15, 23, 30, 50], and image e”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, background): “estimation tasks are very different from models (filters) learned for other image processing tasks such as object recognition [19].”