SibNet: Sibling Convolutional Encoder for Video Captioning

By Sheng Liu, Zhou Ren, Junsong Yuan. 2018

148 citations recorded by the source corpus.

Visual captioning, the task of describing an image or a video using one or few sentences, is a challenging task owing to the complexity of understanding the copious visual information and describing it using natural language. Motivated by the success of applying neural networks for machine translation, previous work applies sequence to sequence learning to translate videos into sentences. In this work, different from previous work that encodes visual information using a single flow, we introduce a novel Sibling Convolutional Encoder (SibNet) for visual captioning, which employs a dual-branch architecture to collaboratively encode videos. The first content branch encodes visual content information of the video with an autoencoder, capturing the visual appearance information of the video as other networks often do. While the second semantic branch encodes semantic information of the video via visual-semantic joint embedding, which brings complementary representation by considering the semantics when extracting features from videos. Then both branches are effectively combined with soft-attention mechanism and finally fed into a RNN decoder to generate captions. With our SibNet explicitly capturing both content and semantic information, the proposed model can better represent the rich information in videos. To validate the advantages of the proposed model, we conduct experiments on two benchmarks for video captioning, YouTube2Text and MSR-VTT. Our results demonstrate that the proposed SibNet consistently outperforms existing methods across different evaluation metrics.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Zicheng Liu
3. Lijuan Wang
4. Luc Van Gool
5. Dacheng Tao
6. Cordelia Schmid
7. Mohit Bansal
8. Jianfeng Gao

In Model Architectures

1. Lin Ma
2. Mohit Bansal
3. Xi Li
4. Wenhao Jiang
5. Kevin Qinghong Lin
6. Zhe Gan
7. Zineng Tang
8. Ce Liu

In Natural Language Generation

1. Wenhao Jiang
2. Mohit Bansal
3. Lin Ma
4. Zineng Tang
5. Zhengyuan Yang
6. Wei David Liu
7. Jie Lei
8. Rita Cucchiara

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[72] Sheng Liu, Zhou Ren, and Junsong Yuan.”
• SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, methodology): “Specifically, these methods [12, 27, 30, 33, 59] employ an encoder to refine video representations from a set of fixed video frame features, and a language decoder operates on top of these refined video representations to learn visual-textual alignment for caption generation.”
• Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, unknown): “1 SibNet [21] 2019 GoogleNet - 54.”
• Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “Trained with the decoder loss alone, the SibNet-DL(G) performs slightly inferiorly to our method.”
• DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021, methodology): “SibNet (Liu et al., 2020b) 40.”
• Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “[88] 2020 CNN/GoogLeNet/Inception, RNN/LSTM Video, Text MSVD, MSR-VTT BLEU, ROUGE, CIDEr, METEOR”