Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think!

By Jack Hessel, Lillian Lee. 2020

97 citations recorded by the source corpus.

Modeling expressive cross-modal interactions seems crucial in multimodal tasks, such as visual question answering. However, sometimes high-performing black-box algorithms turn out to be mostly exploiting unimodal signals in the data. We propose a new diagnostic tool, empirical multimodally-additive function projection (EMAP), for isolating whether or not cross-modal interactions improve performance for a given model on a given task. This function projection modifies model predictions so that cross-modal interactions are eliminated, isolating the additive, unimodal structure. For seven image+text classification tasks (on each of which we set new state-of-the-art benchmarks), we find that, in many cases, removing cross-modal interactions results in little to no performance degradation. Surprisingly, this holds even when expressive models, with capacity to consider interactions, otherwise outperform less expressive models; thus, performance improvements, even when present, often cannot be attributed to consideration of cross-modal feature interactions. We hence recommend that researchers in multimodal machine learning report the performance not only of unimodal baselines, but also the EMAP of their best-performing model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “We also compare to an EMAP (Hessel and Lee, 2020) version, i.e. an ensemble of event and inference-only, without interactions between event/inference (needed for logical misalignments ).”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “A common pitfall when training a joint multimodal model is that complex inter-modal interactions can be ignored during learning, in favor of simpler intra-modal interactions [50, 23, 58].”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “Although such an opaque tool is an acceptable beginning to investigate the need for visual context in MMT, they provide rather indirect evidence (Hessel and Lee, 2020).”