What Value Do Explicit High Level Concepts Have in Vision to Language Problems?

By Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony R. Dick, Anton van den Hengel. 2015

463 citations recorded by the source corpus.

Much recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we investigate whether this direct approach succeeds due to, or despite, the fact that it avoids the explicit representation of high-level information. We propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. We achieve the best reported results on both image captioning and VQA on several benchmark datasets, and provide an analysis of the value of explicit high-level concepts in V2L problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, neutral): “4 CAMP [42] - 72.”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, background): “image. Attribute detection could be regarded as a bottom-up attention process outputting words rather than image feature vectors. However, these and other non-attention attribute detection approaches [55,50] do not retain spatial information, treating the detected attributes as a bag of words. In co”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, methodology): “We use the same fatt(·, ·) as with our spatial attention, with its MLP operating over the temporal dimension T .”
Image Captioning with Semantic Attention (2016, background): “Top-down approaches are the “modern” ones, which formulate image captioning as a machine translation problem [31, 2, 5, 36].”
Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, background): “into caption model training for in-domain data has been established by several works (Fang et al., 2015; Wu et al., 2016; Elliot and de Vries, 2015).”
Exploring Visual Relationship for Image Captioning (2018, background): “Recently, in [35,39], semantic attributes are shown to clearly boost image captioning when injected into CNN plus RNN model and such attributes can be further leveraged as semantic attention [40] to enhance image captioning.”
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering (2016, background): “More experimental results and implementation details can be found in the supplementary file.”
Semantic Compositional Networks for Visual Captioning (2016, methodology): “As shown in [49, 54], detecting explicit semantic concepts encoded in an image, and adding LSTM LSTM”
FVQA: Fact-Based Visual Question Answering (2016, methodology): “– Action: The attribute model [65] provides 24 classes of actions of humans or animals, such as walking, jumping, surfing, swimming.”
Boosting Image Captioning with Attributes (2016, background): “Ours is fundamentally different in the way that [38] is as a result of utilizing attributes to model semantic attention to the locally previous words, as opposed to holistically employing attributes as a kind of complementary representations in this work.”
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “Other lines of research generate image captions using semantic information or via a compositional approach (Fang et al., 2015; Gan et al., 2017; Tran et al., 2016; Jia et al., 2015; Wu et al., 2016; You et al., 2016).”
Variational Autoencoder for Deep Learning of Images, Labels and Captions (2016, methodology): “14 State-of-the-art results using extra information Attributes-CNN+LSTM [7] 0.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “the door to enhanced modularity, versatility, and generalization. 2 Related work Our model connects to multiple lines of research, including works about compositionality [14, 36], concept acquisition [34, 78], and neural computation [26, 60, 7], which have explored the 2 incorporation of structural ”
Areas of Attention for Image Captioning (2016, methodology): “pre-trained object detectors [9, 33, 38].”
Know More Say Less: Image Captioning Based on Scene Graphs (2019, methodology): “To address this issue, many methods explicitly exploit semantic features for images [3], [5], [23], [27]–[29].”