Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning

By Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, Yunde Jia. 2019

85 citations recorded by the source corpus.

Video captioning is a challenging task that involves not only visual perception but also syntax representation learning. Recent progress in video captioning has been achieved through visual perception, but syntax representation learning is still under-explored. We propose a novel video captioning approach that takes into account both visual perception and syntax representation learning to generate accurate descriptions of videos. Specifically, we use sentence templates composed of Part-of-Speech (POS) tags to represent the syntax structure of captions, and accordingly, syntax representation learning is performed by directly inferring POS tags from videos. The visual perception is implemented by a mixture model which translates visual cues into lexical words that are conditional on the learned syntactic structure of sentences. Thus, a video captioning task consists of two sub-tasks: video POS tagging and visual cue translation, which are jointly modeled and trained in an end-to-end fashion. Evaluations on three public benchmark datasets demonstrate that our proposed method achieves substantially better performance than the state-of-the-art methods, which validates the superiority of joint modeling of syntax representation learning and visual perception for video captioning.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Zhe Gan
3. Zicheng Liu
4. Cordelia Schmid
5. Lijuan Wang
6. Dacheng Tao
7. Ming-Hsuan Yang
8. Jitendra Malik

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Linjie Li
6. Xiaowei Hu
7. Zhengyuan Yang
8. Ce Liu

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[38] Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, and Yunde Jia.”
• Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, methodology): “With the advent of the encoder-decoder framework, most of the current work is studying how to better use visual features [35, 23, 1, 36, 22, 24] and design elaborate models [34, 2, 28, 13] to generate sentences directly.”
• Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, unknown): “7 POS+VCT [12] 2019 InceptionResnetV2 C3D - 52.”
• Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, methodology): “[19] both directly optimize the decoding part, which are generally easier to perform well on language metrics compared to methods that focus on the encoding part, such as ours; (3) The more advanced features used (IRv2+I3D optical flow for Wang et al .”