Open-book Video Captioning with Retrieve-Copy-Generate Network

By Ziqi Zhang, Zhongang Qi, Chunfen Yuan, Ying Shan, Bing Li, Ying Deng, Weiming Hu. 2021

118 citations recorded by the source corpus.

In this paper, we convert traditional video captioning task into a new paradigm, i.e., Open-book Video Captioning, which generates natural language under the prompts of video-content-relevant sentences, not limited to the video itself. To address the open-book video captioning problem, we propose a novel Retrieve-Copy-Generate network, where a pluggable video-to-text retriever is constructed to retrieve sentences as hints from the training corpus effectively, and a copy-mechanism generator is introduced to extract expressions from multi-retrieved sentences dynamically. The two modules can be trained end-to-end or separately, which is flexible and extensible. Our framework co-ordinates the conventional retrieval-based methods with orthodox encoder-decoder methods, which can not only draw on the diverse expressions in the retrieved sentences but also generate natural and accurate content of the video. Extensive experiments on several benchmark datasets show that our proposed approach surpasses the state-of-the-art performance, indicating the effectiveness and promising of the proposed paradigm in the task of video captioning.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Cordelia Schmid
  7. Luc Van Gool
  8. Jianfeng Gao

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Linjie Li
  6. Zhengyuan Yang
  7. Xiaowei Hu
  8. Ce Liu

In Natural Language Generation

  1. Zhengyuan Yang

Citations in the corpus