Task-Driven Modular Networks for Zero-Shot Compositional Learning

By Senthil Purushwalkam, Maximilian Nickel, Abhinav Kumar Gupta, Marc'Aurelio Ranzato. 2019

227 citations recorded by the source corpus.

One of the hallmarks of human intelligence is the ability to compose learned knowledge into novel concepts which can be recognized without a single training example. In contrast, current state-of-the-art methods require hundreds of training examples for each possible category to build reliable and accurate classifiers. To alleviate this striking difference in efficiency, we propose a task-driven modular architecture for compositional reasoning and sample efficient learning. Our architecture consists of a set of neural network modules, which are small fully connected layers operating in semantic concept space. These modules are configured through a gating function conditioned on the task to produce features representing the compatibility between the input image and the concept under consideration. This enables us to express tasks as a combination of sub-tasks and to generalize to unseen categories by reweighting a set of small modules. Furthermore, the network can be trained efficiently as it is fully differentiable and its modules operate on small sub-spaces. We focus our study on the problem of compositional zero-shot classification of object-attribute categories. We show in our experiments that current evaluation metrics are flawed as they only consider unseen object-attribute pairs. When extending the evaluation to the generalized setting which accounts also for pairs seen during training, we discover that naive baseline methods perform similarly or better than current approaches. However, our modular network is able to outperform all existing approaches on two widely-used benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[182] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc’Aurelio Ranzato.”
Compositional Few-Shot Recognition with Primitive Discovery and Enhancing (2020, background): “[44] decomposes complex attributes into simple ones, and learns their compositions for zeroshot learning.”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “The task-driven modular network (TMN) [110] employs a modular structure to transfer concepts in the high-level semantic spaces of CNNs and extracts features that are related to all members of the input triplet to determine the jointcompatibility among visual features and object attributes.”
Few-shot Sequence Learning with Transformers (2020, background): “Compositional reasoning is arguably an important skill for few-shot learning (Lake, 2019; Purushwalkam et al., 2019).”
Compositional Few-Shot Class-Incremental Learning (2024, methodology): “(Purushwalkam et al., 2019) decomposes visual features to attributes for zero-shot learning, which encourages the visual features to be close to the combination of attribute features.”
Modularity Improves Out-of-Domain Instruction Following (2020, background): “…or domains through module specialization (Hu et al., 2019b; Blukis et al., 2020), and that modular networks often perform well in compositional generalization because of their composable structure (Devin et al., 2017; Andreas et al., 2017; Bahdanau et al., 2019; Purushwalkam et al., 2019).”
Multi-Task Reinforcement Learning with Soft Modularization (2020, background): “al [26] consider zero-shot compositional learning in vision; Wang et.”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “It should also be mentioned how many explicitly modular architectures have been proposed to improve generalization (e.g. 2019)) and data efﬁciency (Purushwalkam et al., 2019).”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “In visual recognition, researchers explored the compositionality of objects and attributes so that the model can generalize to unseen combinations of visual concepts [28, 29, 53, 50, 35, 30, 23, 56, 26].”