MMTM: Multimodal Transfer Module for CNN Fusion

By Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L. Iuzzolino, Kazuhito Koishida. Computer Vision and Pattern Recognition, 2019

399 citations recorded by the source corpus.

In late fusion, each modality is processed in a separate unimodal Convolutional Neural Network (CNN) stream and the scores of each modality are fused at the end. Due to its simplicity, late fusion is still the predominant approach in many state-of-the-art multimodal applications. In this paper, we present a simple neural network module for leveraging the knowledge from multiple modalities in convolutional neural networks. The proposed unit, named Multimodal Transfer Module (MMTM), can be added at different levels of the feature hierarchy, enabling slow modality fusion. Using squeeze and excitation operations, MMTM utilizes the knowledge of multiple modalities to recalibrate the channel-wise features in each CNN stream. Unlike other intermediate fusion methods, the proposed module could be used for feature modality fusion in convolution layers with different spatial dimensions. Another advantage of the proposed method is that it could be added among unimodal branches with minimum changes in the their network architectures, allowing each branch to be initialized with existing pretrained weights. Experimental results show that our framework improves the recognition accuracy of well-known multimodal networks. We demonstrate state-of-the-art or competitive performance on four datasets that span the task domains of dynamic hand gesture recognition, speech enhancement, and action recognition with RGB and body joints.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/0a4adf4e8ce0c5845950f48054612ca36f8a6f7d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

  1. Dong Yu
  2. Zheng-Hua Tan
  3. Andrew Owens
  4. Alexei A. Efros

In Computer Vision and Pattern Recognition

  1. Zheng-Hua Tan
  2. Meng Yu
  3. Dong Yu
  4. Jesper Højvang Jensen
  5. Shi-Xiong Zhang
  6. Yong Xu
  7. Daniel Michelsanti
  8. Li Deng

In Human-Computer Interaction

  1. Dong Yu

Citations in the corpus