Feature Space Optimization for Semantic Video Segmentation

By Abhijit Kundu, Vibhav Vineet, Vladlen Koltun. 2016

194 citations recorded by the source corpus.

We present an approach to long-range spatio-temporal regularization in semantic video segmentation. Temporal regularization in video is challenging because both the camera and the scene may be in motion. Thus Euclidean distance in the space-time volume is not a good proxy for correspondence. We optimize the mapping of pixels to a Euclidean feature space so as to minimize distances between corresponding points. Structured prediction is performed by a dense CRF that operates on the optimized features. Experimental results demonstrate that the presented approach increases the accuracy and temporal consistency of semantic video segmentation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, methodology): “Some state-of-the-art methods such as [17] incorporate long range spatio-temporal regularization to the out-”
Multi-Scale Context Aggregation by Dilated Convolutions (2015, background): “Bottom: the context module increases accuracy when plugged into a front-end + CRF-RNN conﬁguration (Zheng et al., 2015). is both feasible and desirable.”
Playing for Data: Ground Truth from Computer Games (2016, methodology): “[25], which used a bigger ConvNet and analyzed whole video sequences.”
Deep Layer Aggregation (2017, methodology): “es on validation and to compare against existing methods on test. DLA is the best-in-class among methods in the same setting. Method mIoU SegNet [2] 46.4 DeepLab-LFOV [7] 61.6 Dilation8 [45] 65.3 FSO [24] 66.1 DLA-34 8s 66.7 DLA-34 2s 68.6 DLA-102 2s 71.0 Table 5: Evaluation on CamVid. Higher depth ”
Semantic Video Segmentation by Gated Recurrent Flow Propagation (2016, methodology): “Therefore some of the existing approaches to semantic video segmentation [42, 43, 25] rely on single frame models with corresponding variables connected in time using random fields with higher-order potentials, and mostly prespecified parameters.”
Stacked Deconvolutional Network for Semantic Segmentation (2017, background): “Note that, the CamVid dataset sampling in video frames contains temporal information, and some works [39], [40] mine temporal information to aid segmentation results.”
On the Iterative Refinement of Densely Connected Representation Levels for Semantic Segmentation (2018, background): “The contributions in this direction include Conditional Random Fields (CRFs) and its variants (which remain a popular choice) [24, 7, 49], CRFs as Recurrent Neural Networks [49], iterative inference denoising autoencoders [37], convolutional pseudo-priors [45], as well as graph-cuts, watersheds and ”
STFCN: Spatio-Temporal FCN for Semantic Video Segmentation (2016, methodology): “Dilation + FSO [27] has been published recently based on Dilation8 architecture and became state-of-the-art video semantic segmentation method.”
Soccer on Your Tabletop (2018, background): “Similarly, semantic segmentation can provide pixel-level predictions for a large number of classes [52, 27].”
UAVid: A semantic segmentation dataset for UAV imagery (2018, methodology): “are trained with a Nvidia Titan X GPU. To accommodate the networks into 12G GPU memory, TABLE I IOU SCORES FOR DIFFERENT MODELS. PRT STANDS FOR PRE-TRAIN AND FSO STANDS FOR FEATURE SPACE OPTIMIZATION [32]. IOU SCORES ARE REPORTED IN PERCENTAGE AND BEST RESULTS ARE SHOWN IN BOLD. Model Building Tree ”
Gated Feedback Refinement Network for Dense Image Labeling (2017, methodology): “Following [17, 1, 28], we consider 11 larger semantic classes (road, building, sky, tree, sidewalk, car, column-pole, fence, pedestrian, bicyclist, and sign-symbol) for evaluation.”
Deep Feature Flow for Video Recognition (2016, background): “The dense 3D CRF [24] proposes long-range spatial-temporal regularization in semantic video segmentation.”
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2018, unknown): “[62] use 3D dense CRF to get temporally consistent results.”
AutoRally: An Open Platform for Aggressive Autonomous Driving (2019, background): “Recent works have demonstrated the success of deep neural network architectures consisting of multiple CNN layers in solving challenging semantic segmentation problems [39]–[42].”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “To tackle this problem, Kundu et al. (2016) propose a method for optimizing the feature space of a dense CRF for spatiotemporal regularization.”
Survey on semantic segmentation using deep learning techniques (2019, methodology): “Kundu et al. [147] proposed a model to optimize the feature space used by the fully connected conditional random ﬁeld for spatio-temporal regularization.”
ICNet for Real-Time Semantic Segmentation on High-Resolution Images (2017, background): “computation. Recent Clockwork [29] reuses feature maps given stable video input. Deep feature ow [30] is based on a small-scale optical ow network to propagate features from key frames to others. FSO [31] performs structured prediction with dense CRF applied on optimized features to get temporal con”
A Survey on Deep Learning Technique for Video Segmentation (2021, background): “Later, [220] jointly learns CNN-based per-frame segmentation and CRFbased spatio-temporal reasoning.”