RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes
By Yuxiang Sun, Weixun Zuo, Ming Liu. 2019
472 citations recorded by the source corpus.
Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder–decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Self-Training for Class-Incremental Semantic Segmentation (2020, methodology): “RTFNet [26] fuses both RGB and thermal information to perform semantic segmentation Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
- First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, methodology): “With the video dataset, we analyze fusing three modalities with DeepLabV3+ [13] and benchmark five different state-of-the-art segmentation methods [13, 49, 50, 51, 61].”
- Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “IV-A, there are fewer works on multi-modal semantic segmentation: [91], [118], [123] employ RGB and thermal images, [60] fuses RGB images and depth images from a stereo camera, [124]–[126] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [122]”
- Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs (2019, methodology): “Note that RTFNet is initially designed for RGB and thermal data fusion, but we find that it generalizes well to RGB-D data.”
- Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, methodology): “Most of existing methods mostly develop various simple fusion strategies, such as the weighted average [25, 26, 27], summation [28, 29] and concatenation [30, 31].”
- Deep Multimodal Data Fusion (2024, background): “For example, [69] and [171] did not simply stack depth image (or thermal image) with RGB image to form a four channels input data.”
- Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms (2021, methodology): “Then, we implement our DFM-RTFNet by integrating DFM into RTFNet50 [29], as shown in Fig.”
- CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, background): “RGB-Depth [55], [56], [57] and RGBThermal [58], [59], [18] semantic segmentation are broadly investigated.”