Learning Visual Question Answering by Bootstrapping Hard Attention

By Mateusz Malinowski, Carl Doersch, Adam Santoro, Peter W. Battaglia. 2018

104 citations recorded by the source corpus.

Attention mechanisms in biological perception are thought to select subsets of perceptual information for more sophisticated processing which would be prohibitive to perform on all sensory inputs. In computer vision, however, there has been relatively little exploration of hard attention, where some information is selectively ignored, in spite of the success of soft attention, where information is re-weighted and aggregated, but never filtered out. Here, we introduce a new approach for hard attention and find it achieves very competitive performance on a recently-released visual question answering datasets, equalling and in some cases surpassing similar soft attention architectures while entirely ignoring some features. Even though the hard attention mechanism is thought to be non-differentiable, we found that the feature magnitudes correlate with semantic relevance, and provide a useful signal for our mechanism’s attentional selection criterion. Because hard attention selects important features of the input information, it can also be more efficient than analogous soft attention mechanisms. This is especially important for recent approaches that use non-local pairwise operations, whereby computational and memory costs are quadratic in the size of the set of features.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “…in the form of the language (Ling et al., 2017; Hendricks et al., 2016), attention (Bahdanau et al., 2014; Mnih et al., 2014; Gulcehre et al., 2016; Malinowski et al., 2018; Xu and Saenko, 2016; Yang et al., 2016), phrase selection (Lei et al., 2016), distillation into programs (Hajipour et al.,…”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “…2017b; Perez et al., 2018; Hu et al., 2017; Mascharka et al., 2018; Hu et al., 2018; Suarez et al., 2018) and attention-or memory-based meth-ods (Santoro et al., 2017; Zhu et al., 2017; Hud-son and Manning, 2018; Tan and Bansal, 2018; Malinowski et al., 2018; Pavez et al., 2018; Yao et al., 2018).”
Inferring and Executing Programs for Visual Reasoning (2017, background): “These include for example new attention mechanism [5] and new conditioning methods [6].”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “The second agent does away with the query mechanism entirely and forms the weights for the attention by computing the L2 norm of each key (similar to a soft version of [29]).”
Learning by Abstraction: The Neural State Machine (2019, methodology): “24.96 HAN [59] 28.65 GVQA [3] 31.30 RAMEN [73] 39.21 BAN [46] 39.31 MuRel [15] 39.54 ReGAT [52] 40.42 NSM 45.80 First, we measure the performance on VQA-CP [3], which provides a new split of the VQA2 dataset [27], where the answer distribution is kept different between the training and the test…”
MUREL: Multimodal Relational Reasoning for Visual Question Answering (2019, background): “The soft attention framework is questioned in [27], where regions are hardly selected based on the norm of their feature.”
Variable-rate discrete representation learning (2021, background): “Other models operate on ﬁxed-quantum inputs, but adaptively allocate capacity 5 (Eslami et al., 2016; Graves, 2016; Yeung et al., 2016; Campos et al., 2017; Fig-urnov et al., 2017; Malinowski et al., 2018; Marin et al., 2019; Kirillov et al., 2020; Lugosch et al., 2020).”
MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond (2021, methodology): “Speciﬁcally, we compute the normalized (cid:96) 2 -norm map per-location on the feature map (Malinowski et al., 2018).”
Visual Grounding in Video for Unsupervised Word Translation (2020, background): “Early work explored weak supervision and the correspondence problem between text annotations and image regions [7, 14], with more modern approaches exploring joint image-text word embeddings [17], or building a language conditioned attention map over the images in caption generation, visual question”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, background): “In lieu of directly using visual features from CNN-based feature extractors, [56, 11, 41, 33, 49, 38, 63, 36] explored various image attention mechanisms to locate regions that are relevant to the question.”
VQA With No Questions-Answers Training (2018, background): “1 HAN [50] 99.”
A General Survey on Attention Mechanisms in Deep Learning (2022, methodology): “For example, hard attention is used in [59] for the task of visual question”
Answer Them All! Toward Universal Visual Question Answering Models (2019, background): “There have been recent attempts at reducing the number of pairwise comparisons by reducing the number of input objects fed to RN [34, 1].”
Deep reinforcement learning with relational inductive biases (2018, background): “filtering out unimportant relations (Malinowski et al., 2018).”
Deep Multimodal Data Fusion (2024, methodology): “Similarly, Malinowski et al. [115] propose a hard-attention-based multimodal fusion method, which produces a binary mask over spatial locations determining which features are passed on to down-stream processing.”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “2) HAN [28], GVQA [3], ReGAT [25], NSM [20].”
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning (2019, methodology): “As suggested in (Malinowski et al., 2018), we visualize the l 2 -norm of the last layer features from our 32-frame stride-8 non-local model on the center video crop.”
Reasoning Visual Dialogs With Structural and Partial Observations (2019, methodology): “With the availability of largescale datasets [49, 2, 16, 20, 23], numerous VQA models were proposed to build multimodal representations upon the CNN-RNN architecture [16, 49], and recently extended with differentiable attentions [63, 36, 64, 66, 1, 38].”