Deep End2End Voxel2Voxel Prediction

By Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri. 2015

121 citations recorded by the source corpus.

Over the last few years deep learning methods have emerged as one of the most prominent approaches for video analysis. However, so far their most successful applications have been in the area of video classification and detection, i.e., problems involving the prediction of a single class label or a handful of output variables per video. Furthermore, while deep networks are commonly recognized as the best models to use in these domains, there is a widespread perception that in order to yield successful results they often require time-consuming architecture search, manual tweaking of parameters and computationally intensive preprocessing or post-processing methods. In this paper we challenge these views by presenting a deep 3D convolutional architecture trained end to end to perform voxel-level prediction, i.e., to output a variable at every voxel of the video. Most importantly, we show that the same exact architecture can be used to achieve competitive results on three widely different voxel-prediction tasks: video semantic segmentation, optical flow estimation, and video coloring. The three networks learned on these problems are trained from raw video without any form of preprocessing and their outputs do not require post-processing to achieve outstanding performance. Thus, they offer an efficient alternative to traditional and much more computationally expensive methods in these video domains.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, neutral): “We compare the results to the recently proposed method for video segmentation of [34], which reports results of their architecture with 2D and 3D convolutions.”
Flexible Neural Representation for Physics Prediction (2018, background): “Representations using 3D convolution or point clouds are better at maintaining object shape [46, 47, 10, 36, 37], but do not entirely capture object permanence, and can be computationally inefﬁcient.”
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation (2016, background): “Up-convolutional architectures like the fully convolutional networks for semantic segmentation [8] and the u-net are still not wide spread and we know of only one attempt to generalize such an architecture to 3D [14].”
Optical Flow Estimation Using a Spatial Pyramid Network (2016, methodology): “An alternative approach abandons the classical formulation altogether and starts over using recent neural network architectures [17, 42].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Our research is inspired by the success of applying deep learning to optical ﬂow estimation [2, 12, 16, 19, 50, 51, 52], artistic style transfer [17, 22, 28], and image enhancement [6, 9, 10, 46, 47, 55, 57, 62, 65].”
Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification (2016, methodology): “Inspired by the recently proposed C3D method [9, 8] to learn 3d convolution and deconvolution networks, we designed our networks to learn a new feature representation of videos by exploiting a novel way of training two stream 3D-CNNs.”
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos (2017, methodology): “In pixel-level semantic segmentation, deconvolution proves to be an effective upsampling method in both image [34, 45] and video [61] for producing output of the same resolution as the input.”
Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges (2019, result): “However, there were some experiments that trained from scratch which also delivered better results compared to fine-tuning a pre-trained network [75].”
Two-Stream Convolutional Networks for Dynamic Texture Synthesis (2017, unknown): “[43].”
Video Frame Interpolation via Adaptive Convolution (2017, methodology): “Our work is inspired by the success of deep learning in solving not only difficult visual understanding problems [16, 20, 26, 28, 39, 40, 42, 45, 54, 60, 64] but also other computer vision problems like optical flow estimation [9, 14, 19, 48, 49, 52], style transfer [11, 15, 23, 30, 50], and image e”
Efficient Action Detection in Untrimmed Videos via Multi-task Learning (2016, background): “3D ConvNets have shown to be effective at learning spatiotemporal features in video volume data analysis problems [33] such as video understanding, optical flow estimation, video coloring, etc.”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “Will extra guidance help? Supervised optical flow learning methods [19, 38, 28, 5] showed that using optical flow from traditional methods (we called them proxy ground truths) can help CNNs to learn to predict motions.”
Convolutional Neural Networks for Steady Flow Approximation (2016, methodology): “Recently, CNNs are applied on voxel to voxel prediction problems, such as video coloring [28].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, background): “proposed an U-shape convolution neural network for video colorization [160].”
A survey on deep learning techniques for image and video semantic segmentation (2018, methodology): “Dynamic Graph CNN (DGCNN)[94] Own EdgeConv ✗ ✗ ✗ ✗ EdgeConvolution module for point clouds as graphs Clockwork Convnet[95] FCN ✗ ✗ ✗ ✓ Clockwork scheduling for sequences 3DCNN-Zhang Own 3DCNN ✗ ✗ ✗ ✓ 3D convolutions and graph cut for sequences End2End Vox2Vox[96] C3D ✗ ✗ ✗ ✗ 3D convolutions/deconvol”
Stacked Deconvolutional Network for Semantic Segmentation (2017, methodology): “Specially, our model, without using any temporal, performs better than the models [39], [43] which exploit spatio-temporal relationships between video frames.”
VoxResNet: Deep Voxelwise Residual Networks for Volumetric Brain Segmentation (2016, methodology): “Our method was implemented using Matlab and C++ based on Caffe library [13, 34].”
A Review on Deep Learning Techniques Applied to Semantic Segmentation (2017, methodology): “Another remarkable method, which builds on the idea of using 3D convolutions, is the deep end-to-end voxel-to-voxel prediction system by Tran et al. [89].”
Flow-Guided Feature Aggregation for Video Object Detection (2017, methodology): “On the other hand, exhaustive spatial-temporal convolution is used to directly extract spatial-temporal features [38, 21, 41, 42].”
VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images (2017, unknown): “Our method was implemented using Matlab and C++ based on Caffe library (Jia et al., 2014; Tran et al., 2016).”