SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning

By Kevin Qinghong Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, Lijuan Wang. 2021

341 citations recorded by the source corpus.

The canonical approach to video captioning dictates a caption generation model to learn from offline-extracted dense video features. These feature extractors usually operate on video frames sampled at a fixed frame rate and are often trained on image/video understanding tasks, without adaption to video captioning data. In this work, we present SwinBERT, an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description. Instead of leveraging multiple 2D/3D feature extractors, our method adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates. Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames as opposed to previous successes with sparsely sampled video frames for video-and-language understanding tasks (e.g., video question answering). Moreover, to avoid the inherent redundancy in consecutive video frames, we propose adaptively learning a sparse attention mask and optimizing it for task-specific performance improvement through better long-range video sequence modeling. Through extensive experiments on 5 video captioning datasets, we show that Swinbert achieves across-the-board performance improvements over previous methods, often by a large margin. The learned sparse attention masks in addition push the limit to new state of the arts, and can be transferred between different video lengths and between different datasets. Code is available at https://github.com/microsoft/SwinBERT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Cordelia Schmid
  2. Zhe Gan
  3. Zicheng Liu
  4. Lijuan Wang
  5. Ivan Laptev
  6. Jianfeng Wang
  7. Josef Sivic
  8. Linjie Li

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Cordelia Schmid
  3. Ming-Hsuan Yang
  4. Dacheng Tao
  5. Andrew Zisserman
  6. Jitendra Malik
  7. Trevor Darrell
  8. Xiaogang Wang

In Multimodal Learning

  1. Zhe Gan
  2. Cordelia Schmid
  3. Lijuan Wang
  4. Linjie Li
  5. Zicheng Liu
  6. Jianfeng Wang
  7. Xiaowei Hu
  8. Josef Sivic

In Model Architectures

  1. Aliaksandr Siarohin
  2. Jordi Pont-Tuset
  3. Kevin Qinghong Lin
  4. Ivan Laptev
  5. Jian Ren
  6. Paul Hongsuck Seo
  7. Zhe Gan
  8. Sergey Tulyakov

Citations in the corpus