Unpaired Multi-Modal Segmentation via Knowledge Distillation

By Qi-Yan Dou, Quande Liu, P. Heng, Ben Glocker. 2020

216 citations recorded by the source corpus.

Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model, we introduce a novel loss term inspired by knowledge distillation, by explicitly constraining the KL-divergence of our derived prediction distributions between modalities. We have extensively validated our approach on two multi-class segmentation problems: i) cardiac structure segmentation, and ii) abdominal organ segmentation. Different network settings, i.e., 2D dilated network and 3D U-net, are utilized to investigate our method’s general efficacy. Experimental results on both tasks demonstrate that our novel multi-modal learning scheme consistently outperforms single-modal training and previous multi-modal approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Yuang Liu
6. Jun Wang
7. Tao Lei
8. Asoke Kumar Nandi

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jianping Gou
3. S. Maybank
4. Baosheng Yu
5. Yuang Liu
6. Jun Wang
7. Tao Lei
8. Asoke Kumar Nandi

Citations in the corpus

• Knowledge Distillation: A Survey (2020, background): “…al. 2019; Wu et al. 2020; Wang et al. 2017; Zhang et al. 2020b; Wang et al. 2020b), image/video segmentation (He et al. 2019; Mullapudi et al. 2019; Dou et al. 2020; Liu et al. 2019h; Siam et al. 2019; Hou et al. 2020; Bergmann et al. 2020), action recognition (Luo et al. 2018; Hao and Zhang…”
• Bidirectional Copy-Paste for Semi-Supervised Medical Image Segmentation (2023, unknown): “The first category designed various 2D/3D segmentation network architectures [3, 4, 13, 18, 20, 49].”
• Source-Free Domain Adaptation for Semantic Segmentation (2021, background): “Since then, a variety of KD methods has been presented for model compression [28, 2, 50, 32], domain adaptation [52, 53], and multi-modal learning [51, 14, 10].”
• Medical Image Segmentation Using Deep Learning: A Survey (2020, background): “[149] proposed a novel multi-modal learning scheme for accurate segmentation of”