Deep Multimodal Fusion by Channel Exchanging

By Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, Junzhou Huang. 2020

310 citations recorded by the source corpus.

Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications. Yet, current methods including aggregation-based and alignment-based fusion are still inadequate in balancing the trade-off between inter-modal fusion and intra-modal processing, incurring a bottleneck of performance improvement. To this end, this paper proposes Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Multimodal Data Fusion (2024, background): “Differently, Wang et al. [187] propose a channel-exchanging-network, which dynamically exchanges channels in different modal sub-networks.”
Changer: Feature Interaction is What You Need for Change Detection (2022, background): “For the ﬁrst issue, there are many studies illustrating the important effects of interaction be-tween homo/hetero-geneous features [39, 46].”
Channel-wise Knowledge Distillation for Dense Prediction* (2020, background): “Channel exchanging [33] proposes that the information contained in each channel is general and can be shared across different modalities.”
What Makes Multimodal Learning Better than Single (Provably) (2021, background): “For instance, in aggregation-based multi-modal fusion, g is an aggregation function compounding on K seperate sub-networks and h is a multi-layer neural network [44].”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “Additional methods dynamically exchange channels and shift pixels to enable implicit cross-modal feature fusion [69], [70].”