STFCN: Spatio-Temporal FCN for Semantic Video Segmentation
By Mohammad Hajizadeh Saffar, Mohammad Sabokrou, Mahmood Fathy, Reinhard Klette. 2016
57 citations recorded by the source corpus.
This paper presents a novel method to involve both spatial and temporal features for semantic video segmentation. Current work on convolutional neural networks(CNNs) has shown that CNNs provide advanced spatial features supporting a very good performance of solutions for both image and video analysis, especially for the semantic segmentation task. We investigate how involving temporal features also has a good effect on segmenting video data. We propose a module based on a long short-term memory (LSTM) architecture of a recurrent neural network for interpreting the temporal characteristics of video frames over time. Our system takes as input frames of a video and produces a correspondingly-sized output; for segmenting the video our method combines the use of three components: First, the regional spatial features of frames are extracted using a CNN; then, using LSTM the temporal features are added; finally, by deconvolving the spatio-temporal features we produce pixel-wise predictions. Our key insight is to build spatio-temporal convolutional networks (spatio-temporal CNNs) that have an end-to-end architecture for semantic video segmentation. We adapted fully some known convolutional network architectures (such as FCN-AlexNet and FCN-VGG16), and dilated convolution into our spatio-temporal CNNs. Our spatio-temporal CNNs achieve state-of-the-art semantic segmentation, as demonstrated for the Camvid and NYUDv2 datasets.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Alan L. Yuille
- Chunhua Shen
- Xiaogang Wang
- Ming-Hsuan Yang
- Dacheng Tao
- Jitendra Malik
- Andrew Zisserman
Citations in the corpus
- Occluded Video Instance Segmentation: A Benchmark (2021, background): “LSTM [21], GRU [59], and optical flow [95] are introduced to leverage temporal contextual information for more accurate or faster video semantic segmentation.”
- Deep Feature Flow for Video Recognition (2016, background): “STFCN [10] considers a spatial-temporal FCN for semantic video segmentation.”
- Survey on semantic segmentation using deep learning techniques (2019, methodology): “Fayyaz et al. [145] presented a full convolutional network Spatio-Temporal Fully Convolutional Network (STFCN) employing spatial and temporal features.”
- Flow-Guided Feature Aggregation for Video Object Detection (2017, background): “On one hand, most of these work [34, 24, 47, 7, 46, 1, 9, 35] use recurrent neural network (RNNs) to aggregate features from consecutive frames.”
- Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges (2017, methodology): “It is worth noting that in case of FCN8s, we were able to reproduce similar results to the work in[32].”