Predicting Deeper into the Future of Semantic Segmentation

By Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, Yann LeCun. 2017

255 citations recorded by the source corpus.

The ability to predict and therefore to anticipate the future is an important attribute of intelligence. It is also of utmost importance in real-time systems, e.g. in robotics or autonomous driving, which depend on visual scene understanding for decision making. While prediction of the raw RGB pixel values in future video frames has been studied in previous work, here we introduce the novel task of predicting semantic segmentations of future frames. Given a sequence of video frames, our goal is to predict segmentation maps of not yet observed video frames that lie up to a second or further in the future. We develop an autoregressive convolutional neural network that learns to iteratively generate multiple frames. Our results on the Cityscapes dataset show that directly predicting future segmentations is substantially better than predicting and then segmenting future RGB frames. Prediction results up to half a second in the future are visually convincing and are much more accurate than those of a baseline based on warping semantic segmentations using optical flow.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VideoGPT: Video Generation using VQ-VAE and Transformers (2021, methodology): “Architectures developed for video generation can be useful in forecasting applications for weather prediction (Sønderby et al., 2020), autonomous driving (for e.g., such as predicting the future in more semantic and dense abstractions like segmentation masks (Luc et al., 2017)).”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, neutral): “One can also use the temporal structure in a video for self-supervised methods [21, 27, 41, 46, 47].”
Deep Learning: A Critical Appraisal (2018, background): “In interviews, deep learning pioneers Geoff Hinton and Yann LeCun have both recently pointed to unsupervised learning as one key way in which to go beyond supervised, data-hungry versions of deep learning.”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, background): “[2], other semi-supervised self-training in semantic segmentation [30, 31, 32, 33, 34], and other approaches based on pseudo label such as [4, 35, 36, 3].”
Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning (2017, background): “Other approaches for prediction try to build an autoregressive model that predicts future segmentations given visual segments from a perception layer [Neverova et al., 2017].”
QuaterNet: A Quaternion-based Recurrent Model for Human Motion (2018, background): “In terms of neural generative modeling, adversarial networks [35, 38, 56, 59] and variational auto-encoders [58, 59] are explored to explicitly deal with the intrinsic uncertainty of forecasting tasks.”
ClusterFit: Improving Generalization of Visual Representations (2019, background): “…pre-text tasks can either be domain agnostic [5, 6, 29, 45, 60, 61] or exploit domain-speciﬁc information like spatial structure in images [13, 21, 42–44], color [12, 34, 35, 63, 64], illumination [14], temporal structure [16, 25, 37, 39, 40] or a co-occurring modality like sound [2, 3, 9, 19, 46].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, background): “[41] predicts videos of semantic maps and argues that autoregressive models lead to error propagation when more frames are predicted while being more accurate in the semantic segmentation space.”
Improving Semantic Segmentation via Video Propagation and Label Relaxation (2018, methodology): “[28] use a state-of-theart image segmentation method [42] as a teacher to generate extra annotations for unlabelled images.”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “An improvement on short-term predictions were reported over previous works [56], [175], yet performing worse on midterm predictions.”
Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods (2018, methodology): “More recently, Luc et al. (2018) has extended the model of Luc et al. (2017) to the related task of future instance segmentation prediction.”
Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation (2021, background): “Semantic segmentation aims to predict semantic labels of each pixel in the given images, which plays an important role in autonomous driving [19] and medical diagnosis [28].”
When will you do what? - Anticipating Temporal Occurrences of Activities (2018, background): “Predicting future frames, poses or image segmentations in videos has been studied in several works [20, 16, 13, 31, 11, 27, 18, 29, 33, 32, 17, 12].”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, background): “Human-annotated ground-truth labels of certain frames in a video sequence could be propagated to other unlabeled frames via patch matching [4,6] or optical flow [54,23,94,53,56].”
Predicting Video with VQVAE (2021, background): “Finally, some have proposed a hierarchical approach based on structured information—generating video conditioned on text [25], semantic segments [29, 28], or human pose [56, 53].”