MAttNet: Modular Attention Network for Referring Expression Comprehension

By Licheng Yu, Zhe L. Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L. Berg. 2018

966 citations recorded by the source corpus.

In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit, we propose to decompose them into three modular components related to subject appearance, location, and relationship to other objects. This allows us to flexibly adapt to expressions containing different types of information in an end-to-end framework. In our model, which we call the Modular Attention Network (MAttNet), two types of attention are utilized: language-based attention that learns the module weights as well as the word/phrase attention that each module should focus on; and visual attention that allows the subject and relationship modules to focus on relevant image components. Module weights combine scores from all three modules dynamically to output an overall score. Experiments show that MAttNet outperforms previous state-of-the-art methods by a large margin on both bounding-box-level and pixel-level comprehension tasks. Demo1 and code2 are provided.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “Thus we directly use the bounding box proposals provided by [33], which use a Mask R-CNN [34] pretrained on the COCO dataset.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “In this paper, we use the bounding box proposals provided by [35] pre-trained on the MS-COCO dataset.”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “We adopt the context-query attention mechanism from the question answering literature (Yu et al., 2018a) to aggregate the two representations h o t and h g .”
Unifying Vision-and-Language Tasks via Text Generation (2021, methodology): “…performance, these results suggest that referring expression comprehension can be effectively formulated as a text-generation task, rather than previously (Yu et al., 2018a; Chen et al., 2020) formulated classiﬁcation task over a set of visual regions, allowing more ﬂexible architecture design.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “(6) We use a combined hinge loss Lh (Yu et al., 2018) over positive and negative query-video pairs.”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, methodology): “Model training and evaluation are conducted either on the groundtruth RoIs or on the detected boxes in MAttNet (Yu et al., 2018).”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, neutral): “Different from earlier work [16,22,28,29,50,51,55] which design bespoke architecture for different tasks, recently proposed models for V&L [1, 8, 23, 24, 27, 43, 45, 54] provide a common architecture that can be pretrained using self-supervised losses and adapted to many vision and language tasks.”
TVQA: Localized, Compositional Video Question Answering (2018, methodology): “The context matching module is adopted from the context-query attention layer from previous works (Seo et al., 2017; Yu et al., 2018a).”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “…we sample two negative pairs of ( q i , v j ) and ( q z , v i ) for each positive pair of ( q i , v i ) to calculate a combined hinge loss as [37]: Single Video Moment Retrieval Given the START and END probabilities P st , P ed , we deﬁne the single video moment retrieval loss as: where t i…”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “Speciﬁcally, we use cross-entropy loss to supervise the learning of the start and end index for local alignment and a combined hinge loss [76] over positive and negative query-video pairs for global alignment.”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “As vision and language usually contain complementary information, joint learning of vision and language representations has been proven to be quite effective on multimodality tasks, such as Visual Question Answering [1, 2, 18, 33], Image Captioning [29, 67], and Referring Expression [68].”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, background): “, 2020), Referring Expressions (Yu et al., 2018) and Visual Commonsense Reasoning (Zellers et al.”
PointCLIP: Point Cloud Understanding by CLIP (2021, methodology): “Therein, multiview methods project point clouds onto images of multiple views and process them with 2D Convolution Neural Networks (CNN) [22] pre-trained on ImageNet [28], such as MVCNN [49] and others [14, 15, 21, 26, 62].”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, neutral): “[63] and Huang et al .”