ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

By Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda G. Shapiro, Hannaneh Hajishirzi. 2018

898 citations recorded by the source corpus.

We introduce a fast and efficient convolutional neural network, ESPNet, for semantic segmentation of high resolution images under resource constraints. ESPNet is based on a new convolutional module, efficient spatial pyramid (ESP), which is efficient in terms of computation, memory, and power. ESPNet is 22 times faster (on a standard GPU) and 180 times smaller than the state-of-the-art semantic segmentation network PSPNet, while its category-wise accuracy is only 8% less. We evaluated EPSNet on a variety of semantic segmentation datasets including Cityscapes, PASCAL VOC, and a breast biopsy whole slide image dataset. Under the same constraints on memory and computation, ESPNet outperforms all the current efficient CNN networks such as MobileNet, ShuffleNet, and ENet on both standard metrics and our newly introduced performance metrics that measure efficiency on edge devices. Our network can process high resolution images at a rate of 112 and 9 frames per second on a standard GPU and edge device, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Another line of research [15, 61, 94] aimed at fusing features from different receptive fields.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, methodology): “Atrous convolution [12, 48] and encoder-decoder architecture [57, 50, 2] are the most common methods for retaining spatial information.”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, neutral): “DeepLab [5, 6] and its follow-ups [8, 54, 39] adopt dilated convolutions and fuse different dilation rate features to increase the receptive ﬁled of the network.”
Graph-Guided Architecture Search for Real-Time Semantic Segmentation (2019, neutral): “1 ESPNet [28] 1024x512 60.”
Partial Order Pruning: For Best Speed/Accuracy Trade-Off in Neural Architecture Search (2019, neutral): “ESPNet [16] - 60.”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, background): “Otherwise, weights are applied to padded zeros instead of the valid spatial region (Yu & Koltun, 2016; Chen et al., 2017; Mehta et al., 2018).”
Structured Knowledge Distillation for Semantic Segmentation (2019, methodology): “Blue circles are collected from FCN* [38], RefineNet [23], SegNet [3], ENet [31], PSPNet [56], ERFNet [34], ESPNet [28], MobileNetV2Plus [25], and OCNet [50].”
In Defense of Pre-Trained ImageNet Architectures for Real-Time Semantic Segmentation of Road-Driving Images (2019, background): “Many real-time semantic segmentation approaches [28, 40, 24, 32] address this goal by introducing custom lightweight architectures which are not suited for largescale visual recognition.”
Fast-SCNN: Fast Semantic Segmentation Network (2019, background): “We discuss and compare semantic image segmentation frameworks with a particular focus on real-time execution with low energy and memory requirements [2, 20, 21, 36, 34, 17, 25, 18].”
All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda (2021, methodology): “Theses methods explore the image crop/resizing [367] or efficient network design [368], [369] or transfer learning [370], [371].”
Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms (2021, methodology): “ESPNet [27], GSCNN [26], and DeepLabv3+ [10] are single-modal networks, while FuseNet [28], RTFNet [29], and our DFM-RTFNet are data-fusion networks.”
MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification (2020, neutral): “, [23, 21, 6, 26].”
CGNet: A Light-Weight Context Guided Network for Semantic Segmentation (2018, methodology): “Furthermore, the accuracy of our approach is 4.5% higher than the very recent model ESPNet [21].”
Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild (2021, background): “networks, ENet [31], SQNet [32], ERFNet [33], ICNet [16], LinkNet [34], ESPNet [35], EDANet [36], BiSeNet [37], CGNet [38] and SwiftNet [39].”
BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation (2020, methodology): “Meanwhile, ESP-Net (Mehta et al., 2018, 2019) devises an eﬃcient spatial pyramid dilated convolution for real-time semantic segmentation.”
Survey on semantic segmentation using deep learning techniques (2019, background): “For example, Inception [27], Xception [31], ResNet [34], ASP [99], ESP [47]; Shu ﬄ eNet [223] and MobileNet [224], are using grouped and depth-wise convolutions.”