Coherent Multi-sentence Video Description with Variable Level of Detail

By Anna Rohrbach, Marcus Rohrbach, Weijian Qiu, Annemarie Friedrich, Manfred Pinkal, Bernt Schiele. 2014

230 citations recorded by the source corpus.

Humans can easily describe what they see in a coherent way and at varying level of detail. However, existing approaches for automatic video description focus on generating only single sentences and are not able to vary the descriptions’ level of detail. In this paper, we address both of these limitations: for a variable level of detail we produce coherent multi-sentence descriptions of complex videos. To understand the difference between detailed and short descriptions, we collect and analyze a video description corpus of three levels of detail. We follow a two-step approach where we first learn to predict a semantic representation (SR) from video and then generate natural language descriptions from it. For our multi-sentence descriptions we model across-sentence consistency at the level of the SR by enforcing a consistent topic. Human judges rate our descriptions as more readable, correct, and relevant than related work.

Subfields: Computer Vision and Pattern Recognition, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Yi Yang
  6. Andrew Zisserman
  7. Jitendra Malik
  8. Ming-Hsuan Yang

In Natural Language Generation

  1. Kate Saenko
  2. Marcus Rohrbach
  3. Lisa Anne Hendricks
  4. Subhashini Venugopalan
  5. Trevor Darrell
  6. Jeff Donahue
  7. Peter Anderson
  8. Anna Rohrbach

Citations in the corpus