In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images

By Marin Orsic, Ivan Kreso, Petra Bevandic, Sinisa Segvic. Computer Vision and Pattern Recognition, 2019

393 citations recorded by the source corpus.

Recent success of semantic segmentation approaches on demanding road driving datasets has spurred interest in many related application fields. Many of these applications involve real-time prediction on mobile platforms such as cars, drones and various kinds of robots. Real-time setup is challenging due to extraordinary computational complexity involved. Many previous works address the challenge with custom lightweight architectures which decrease computational complexity by reducing depth, width and layer capacity with respect to general purpose architectures. We propose an alternative approach which achieves a significantly better performance across a wide range of computing budgets. First, we rely on a light-weight general purpose architecture as the main recognition engine. Then, we leverage light-weight upsampling with lateral connections as the most cost-effective solution to restore the prediction resolution. Finally, we propose to enlarge the receptive field by fusing shared features at multiple resolutions in a novel fashion. Experiments on several road driving datasets show a substantial advantage of the proposed approach, either with ImageNet pre-trained parameters or when we learn from scratch. Our Cityscapes test submission entitled SwiftNetRN-18 delivers 75.5% MIoU and achieves 39.9 Hz on 1024×2048 images on GTX1080Ti.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/833ae7114af5756a674e914ab9a6e1a188e8df5a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, neutral): “[42] - 75.”
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation (2020, methodology): “GFLOPs and train-able parameters are calculated using the pytorch-OpCounter library [60], also used by others [31].”
BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation (2020, neutral): “2019b), SwiftNet (Orsic et al. 2019), SwiftNet-pyr (Orsic and Segvic 2021), BiSeNetV1 (Yu et al.”
Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes (2023, methodology): “One utilizes the GPU-efﬁcient backbones, especially ResNet-18 [20], [21], [22].”
FaPN: Feature-aligned Pyramid Network for Dense Image Prediction (2021, neutral): “In particular, on Cityspaces, FaPN-R18 runs 2× faster than SwiftNet [38], while maintaining a similar mIoU performance.”
Lite-HRNet: A Lightweight High-Resolution Network (2021, methodology): “02 GFLOPs, outperforming the hand-crafted methods [34, 55, 50, 23, 58, 36, 41, 12] and NAS-based methods [54, 25, 16, 24] , and comparable with SwiftNetRN18 [32] that is far computationally intensive (104 GFLOPs).”
Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes (2021, methodology): “Similar to SPP in SwiftNet [21], DAPPM is implemented with the sequence BN-ReLU-Conv.”
Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild (2021, methodology): “computationally-expensive networks including SegNet [10], PSPNet [14], DenseASPP [3] and DANet [26], as well as 4 efficient networks including CGNet [38], SwiftNet [39], SwaftNet [4] and ERF-PSPNet [1], as shown in Table V.”
BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies (2021, methodology): “GTX 1080 Ti, as proposed by other works for fair comparison [31].”
TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation (2022, methodology): “SwiftNet [35] uses lateral connections as the cost-effective solution to restore the prediction resolution while maintaining the speed.”
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis (2020, methodology): “Our decoder module extends the one of SwiftNet [30], which is comprised of a 3×3 convolution with a fixed number of 128 channels and a subsequent bilinear upsampling.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “Table V shows per-class accuracy of our approach compared to RGB-only [33], [80] and RGB-Polarization fusion methods [12], [56] on ZJU-RGB-P dataset [12].”
Semantic Flow for Fast and Accurate Scene Parsing (2020, methodology): “DFANet [24] utilizes a light-weight backbone to speed up and is equipped with a cross-level feature aggregation to boost accuracy, while SwiftNet [38] uses lateral connections as the cost-effective solution to restore the prediction resolution while maintaining the speed.”
Graph-Guided Architecture Search for Real-Time Semantic Segmentation (2019, neutral): “7 SwiftNet [33] 1024x2048 75.”
PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers (2022, methodology): “Before fine-tuning our models, we pre-train them by ImageNet [41] as most of previous works doing [20,34,35].”
Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels (2020, background): “Recently, a large interest in developing extremely efficient architectures arised [16], [17], [18], [19], [20].”