BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation

By Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, Nong Sang. International Journal of Computer Vision, 2020

1,654 citations recorded by the source corpus.

Low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, leading to a considerable decrease in accuracy. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for real-time semantic segmentation. For this purpose, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves the following: (i) A detail branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) A semantics branch, with narrow channels and deep layers to obtain high-level semantic context. The detail branch has wide channel dimensions and shallow layers, while the semantics branch has narrow channel dimensions and deep layers. Due to the reduction in the channel capacity and the use of a fast-downsampling strategy, the semantics branch is lightweight and can be implemented by any efficient model. We design a guided aggregation layer to enhance mutual connections and fuse both types of feature representation. Moreover, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture shows favorable performance compared to several state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2048×1024\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2048\times 1024$$\end{document} input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy. The code and trained models are available online at https://git.io/BiSeNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6142526105340bfbe17107032543bebcf93bf3ae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lu Yuan
2. Tal Hassner
3. Bin Xiao
4. Jingdong Wang
5. Lei Zhang
6. Lior Wolf
7. Changqian Yu
8. Changxin Gao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jingdong Wang
3. Dacheng Tao
4. Chunhua Shen
5. Ming-Hsuan Yang
6. Xiaogang Wang
7. Yi Yang
8. Lei Zhang

Citations in the corpus

• FaPN: Feature-aligned Pyramid Network for Dense Image Prediction (2021, methodology): “Specifically, FaPN-R34 outperforms BiSeNetV2 [48] in both segmentation accuracy measured in mIoU and inference speed.”
• PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers (2022, methodology): “Following [20, 51], we finetune the Cityscapes pretrained models for CamVid and stop the training process when lr < 5e−4 to avoid overfitting.”
• Rethinking BiSeNet For Real-time Semantic Segmentation (2021, methodology): “BiSeNet [28, 27] has been proved to be a popular twostream network for real-time segmentation.”
• Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes (2023, methodology): “BiSeNetV2 [23] proposes a booster training strategy in which extra segmentation heads are added at the end of each stage of the semantic branch.”
• Lite-HRNet: A Lightweight High-Resolution Network (2021, methodology): “The initial rate is set to 1e with a “poly” learning rate strategy [50, 49] with a multiplier of (1 − iter max iters ) each iteration.”
• Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes (2021, methodology): “The crop size is 640 × 640, as same as that of BiSeNetV2 [24].”
• UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery (2021, positive): “This multi-head segmentation architecture has been demonstrated to be effective in previous research (Yu et al., 2020; Zhu et al., 2019).”
• HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation (2020, methodology): “BiSeNetV2 [51] extends BiSeNet by offering a more elaborate fusion of the two branches and additional prediction heads from intermediate layers for boosting the training.”