MDETR - Modulated Detection for End-to-End Multi-Modal Understanding

By Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, Nicolas Carion. 2021

1,168 citations recorded by the source corpus.

Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes. This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text. In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. We pre-train the network on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. We then fine-tune on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation, achieving state-of-the-art results on popular benchmarks. We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting. We show that our pre-training approach provides a way to handle the long tail of object categories which have very few labelled instances. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR. The code and models are available at https://github.com/ashkamath/mdetr.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Sergey Levine
4. Trevor Darrell
5. Zhe Gan
6. Quoc V. Le
7. Dacheng Tao
8. Cordelia Schmid

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Jitendra Malik
7. Cordelia Schmid
8. Andrew Zisserman

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Jianfeng Gao
4. Linjie Li
5. Zicheng Liu
6. Cordelia Schmid
7. Jianfeng Wang
8. Mohit Bansal

Citations in the corpus

• Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “We may add new objects to the currently visible objects list if, after a robot action, MDETR detects an object that we have not seen before.”
• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Future captioning work with SMs may explore open-vocabulary object detectors [65, 66] to recover more salient details, or combine the outputs of multiple task-specific image captioning models with LMs to assemble a single rich and coherent caption.”
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “Grounding [62] FasterRCNN[39] - sentence partiallabel partiallabel FindIt[21] FasterRCNN[39] A sentence partiallabel fine-tune MDETR[18] DETR[2] A,C word fine-tune zero-shot fine-tune DQ-DETR[46] DETR[2] A,C word zero-shot fine-tune GLIP[26] DyHead[5] A word zero-shot zero-shot zero-shot GLIPv2[59] DyHe”
• Generalized Decoding for Pixel, Image, and Language (2022, background): “[35] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.”
• Language-driven Semantic Segmentation (2022, methodology): “, 2021) introduces an advanced zero-shot object detection method that leverages CLIP, whereas MDETR (Kamath et al., 2021) proposes an end-to-end approach that modulates a transformer-based baseline detector with text features that are obtained from a state-of-the-art language model.”
• Grounded Language-Image Pre-training (2021, methodology): “In visionlanguage literature [8, 21, 29, 30, 32, 38, 50, 52, 67], deep fusion of visual and language features is necessary to learn a performant phrase grounding model.”
• Transformers in Vision: A Survey (2021, background): “Modulated DETR (MDETR) [203] has a CNN and BERT backbone to extract features from image and text inputs, respectively.”
• CLIPort: What and Where Pathways for Robotic Manipulation (2021, methodology): “With the advent of large-scale models [37, 38, 39], a number of methods for learning joint vision and language representations have been proposed [13, 14, 15, 20, 40].”
• Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “Many recent works aim to transfer the language capabilities of these models to object detection [12,26,45,46,20].”
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[40] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel”
• Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “Recently, some state-of-the-art REC methods [8, 22, 25] have shown that transformer can capture intra- and inter-modal contexts homogeneously across visual and textual modalities.”
• V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “Objects365 v2 [42] and GoldG [17] datasets are only used as detection data.”
• Image Segmentation Using Text and Image Prompts (2021, methodology): “MDETR [21] is an adaptation of the detection method DETR [22] to nat-ural language phrase input.”