Meta Module Network for Compositional Visual Reasoning

By Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Yang Wang, Jingjing Liu. 2019

82 citations recorded by the source corpus.

Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical draw-backs: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Prominent work includes: (i) advanced attention mechanisms [40, 79, 51]; (ii) better multimodal fusion methods [14, 80, 27, 26]; (iii) multi-step reasoning [77, 22, 15]; (iv) incorporation of object relations [55, 48, 6, 32]; and (v) neural module networks for compositional reasoning [3, 25, 20, 11]”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “Method LXMERT MMN [5] 12-in-1 NSM [13] OscarB OscarB ⇤ Test-dev 60.00 61.19 61.58 Test-std 60.33 60.83 60.65 63.17 61.23 61.62 (b) VQA Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks 21 Table 8: NoCaps val set.”
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, methodology): “We take our pre-trained model trained for 40 epochs on our combined dataset, and initialise these queries as well as the heads for each of them randomly, and fine-tune first for 125 epochs on the unbalanced all GQA split, followed by 10 epochs on the balanced split similar to what is done in prior w”
VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “Method LXMERT MMN [3] 12-in-1 OSCARB NSM [12] OSCAR+B”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, unknown): “Many neural network models have been proposed for these tasks, such as more advanced attention mechanisms [64, 43, 70], better multimodal fusion methods [15, 71, 31, 30], the use of multi-step reasoning [24, 17, 7], the incorporation of relations [49, 38, 45], and neural module networks for composit”
VinVL: Revisiting Visual Representations in Vision-Language Models (2021, methodology): “Method LXMERT MMN [3] 12-in-1 OSCARB NSM [11] OSCAR+B w/ VINVL Test-dev 60.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Our work could also be regarded as neural module networks [2, 4, 9, 13, 18, 19, 46,66], which provides transparent step-by-step reasoning.”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “These methods can be categorized into three groups: joint learning mechanisms: BAN [26], Pythia [24], DFAF [15], fPMC [20], STL [54], CTI [11], and MCAN [59]; reasoning-based methods: Murel [4], ReGAT [30], MMN [6], NMS [22], and HAN [25]; and largescale visual-language modeling: LXMERT [47], OSCAR ”
SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels (2021, unknown): “Models MMN NSM LXMERT 12-in-1 OSCAR SemVLP (Chen et al., 2019a) (Hudson and Manning, 2019a)”
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, methodology): “Empirical results show that 3LMH [69] and MMN [11] on VQA-CP v2 and GQA are used to plot the SOTA polygon for fair comparison.”