Semantic Compositional Networks for Visual Captioning

By Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, Li Deng. 2016

451 citations recorded by the source corpus.

A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “It also drives research progress in multimodal learning and inference across vision and language, which is one of the most active research areas in recent years [20, 18, 36, 19, 41, 4, 30, 5, 1, 31, 33, 32]”
Attention on Attention for Image Captioning (2019, neutral): “[42, 27, 47, 2, 7, 16, 15] and achieved impressive results.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “There is a natural link between the natural language and visions, and most questions require joint modeling of both two streams to produce the satisfied results [15, 26, 48], e.”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “, via mean-pooling [31, 10], recurrent nets [7, 30], and attention mechanisms [35, 22, 10].”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017), localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017), and video question answering.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “However, Gan et al. [41] introduced a Semantic Compositional Network (SCN) for image captioning.”
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “Other lines of research generate image captions using semantic information or via a compositional approach (Fang et al., 2015; Gan et al., 2017; Tran et al., 2016; Jia et al., 2015; Wu et al., 2016; You et al., 2016).”
CNN+CNN: Convolutional Decoders for Image Captioning (2018, methodology): “Alternatively, [9] uses semantics differently, by generating the parameters of an RNN or LSTM from the output of the semantic detector.”
Video Captioning With Attention-Based LSTM and Semantic Consistency (2017, background): “Due to the development of Recurrent Neural Network (RNN) [29] and LSTM, researchers have striven to automatically describe an image/video with a correct and novel natural language sentence [30], [31], [20], [32], [33], [34], [35], [36], [37], [38].”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, methodology): “Video captioning is a widely studied problem in computer vision [22,23,24,25].”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “Video Captioning: Captioning is crucial for understanding visuals; however it is typically limited to describing observable objects and events (Yang et al., 2011; Thomason et al., 2014; Gan et al., 2017)), or for generating paragraphs or multisentence captions about the image or video (Krause et al.”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “On top of that, methods such as conditional random ﬁelds (CRF) [12] and recurrent neural networks (RNNs) [12, 52] are proposed to replace the original mean-pooling in the encoder, and attention mechanisms [15, 35, 62, 28] were incorporated to extract salient frames or regions in decoding phase.”
Towards Better Representations with Deep/Bayesian Learning (2018, methodology): “In the traditional factor analysis (Gan et al., 2017c), the linear relation between x and z is assumed, and inference is done using mean field VB or Gibbs sampling.”
Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “Most existing methods of video captioning [48, 41, 13, 30, 4, 12, 8, 1] mainly focus on investigating various visual perception models by exploiting informative semantics without considering learning of syntax representation for generating sentences.”
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation (2017, methodology): “We have witnessed the resurgence of image captioning [42, 29, 18, 6, 40, 4, 47, 12, 33, 10] which is often addressed by jointly modeling visual and textual content with deep neural networks.”
From Show to Tell: A Survey on Image Captioning (2021, background): “Other approaches [29], [30] integrated high-level attributes or tags,”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, background): “…numerous improvements have been introduced, such as attention (Yao et al. 2015), hierarchical recurrent neural network (Yu et al. 2016; Pan et al. 2016), multi-modal fusion(Gan et al. 2017; Shen et al. 2017; Wang, Wang, and Wang 2018), multi-task learning (Pasunuru and Bansal 2017), etc.”