TreeTalk: Composition and Compression of Trees for Image Descriptions

By Polina Kuznetsova, Vicente Ordonez, Tamara L. Berg, Yejin Choi. 2014

288 citations recorded by the source corpus.

We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions. We investigate two related tasks: image caption generalization and generation, where the former is an optional subtask of the latter. The high-level idea of our approach is to harvest expressive phrases (as tree fragments) from existing image descriptions, then to compose a new description by selectively combining the extracted (and optionally pruned) tree fragments. Key algorithmic components are tree composition and compression, both integrating tree structure with sequence structure. Our proposed system attains significantly better performance than previous approaches for both image caption generalization and generation. In addition, our work is the first to show the empirical benefit of automatically generalized captions for composing natural image descriptions.

Subfields: Computer Vision and Pattern Recognition, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Yi Yang
  6. Chunhua Shen
  7. Jitendra Malik
  8. Ming-Hsuan Yang

In Natural Language Generation

  1. Marcus Rohrbach
  2. Trevor Darrell
  3. Kate Saenko
  4. Lisa Anne Hendricks
  5. Subhashini Venugopalan
  6. Rita Cucchiara
  7. Lorenzo Baraldi
  8. Marcella Cornia

Citations in the corpus